April 30, 2012
Our internet connection has been up and down for a couple of weeks. This is odd. Since getting Verizon FiOS years ago, we've rarely had a problem. But this one drove me insane.
How It Went Down
Wednesday. After a day or two of random internet outages - fixable by rebooting the router every couple of hours - I called the Verizon tech support people. I described the symptoms and they diagnosed a bad router, citing a crapped-out wireless transmitter. I objected, quietly, since it wasn't just impacting the wireless connection but also wired connections. But it's the first thing my former help desk-running self would have done in their position so, whatever. The router came the next day. I installed it. Same problem.
Monday. Are you there Verizon, it's me Chris. I felt genuinely sorry for the guy I got on the phone. He sounded clueless-by-birth and stumped around this particular issue specifically. Apparently the old router-swap routine rarely fails. After I told him again that particular silver bullet didn't do it's thing, he put me on hold for a very long time. When he returned he said (and I shit you not this is how it played out and since, for quality control purposes this phone call was recorded, somewhere it's on tape):
Him: I have a solution I want you to try.
Me: Okay, let's hear it.
Him: After we get off the phone, I want you to unplug the coax cable from the wall and blow on it. Then put it back.
Me: Blow on it?
Him: Yeah, it might clear the static on the line.
Me: You're kidding.
Him: No, blow on it. And if that doesn't fix the problem, call us back.
Me: Oh, I think you can count on that.
So I blew on it. Five minutes later our internet connection dropped.
Wednesday. I gave my buddies at Verizon another call. They agreed to send someone over on Friday afternoon.
Friday. Five minutes before the Verizon guy arrives, the internet died. I left it that way. You know, you want the mechanic to be able to hear the terrible noise your car is making though it never makes it when you need it to. He verified that there was indeed a problem but couldn't figure out why. The best bet, he said, was replacing the box on the outside of the house where the cable came in.
This was an odd choice since I'd spent a couple of minutes earlier that day checking the log files. Something was clearly up but I couldn't tell exactly what. So I asked:
Me: Do you guys check the log files on routers before you start swapping out equipment?
Him: Nah, not usually. I'm not a computer guy.
And I'm thinking, you're not a computer guy? All this whole thing is is just a bunch of computers talking to each other! But no, he just replaced the hardware outside the house and everything worked fine.
Saturday. I woke up Saturday morning to a world in which no television, phone or internet existed. At least in my house. I got my ass on the phone (the rest of me too) and they promised to send someone out that day. They did. The battery backup unit was dead. They replaced it. Everything is working dandy.
The Actual Problem
The problem was actually pretty simple. Here it is in layman's terms.
Routers have tables - code, really - that direct traffic. Depending on the router you have, sometimes those tables are pretty big, sometimes pretty small. If you're an average user, it doesn't really matter. But if you have a lot of devices, lots of connections to the outside world and a pretty good deal of traffic, you want a router with a big table. When those tables get full, weird things start to happen. Primarily, your internet connection drops and our router has to be rebooted. Verizon happens to give out routers with pretty small tables and doesn't offer any alternative.
Why This Is Annoying
For the thousands of dollars on hardware and manpower they spent on my little problem, they never actually diagnosed it. They patched it with hardware instead of finding out the reason. Know how I know? Because I figured out the problem over the weekend. The answer's Googleable. It's all over and apparently a big enough issue that every single Verizon tech should have asked some different questions. And it had absolutely nothing to do with hardware. As a result they fixed stuff that didn't need fixing, actually broke stuff in the process, and cause a pretty major outage.
This is why your cable and internet bills are so high, people.Posted by Chris at April 30, 2012 7:39 AM