[ EPU Foyer ] [ Lab and Grill ] [ Bonus Theater!! ] [ Rhetorical Questions ] [ CSRANTronix ] [ GNDN ] [ Subterranean Vault ] [ Discussion Forum ] [ Gun of the Week ]

Eyrie Productions, Unlimited

Subject: "Elder Days Story Time: Cablevision Employee"     Previous Topic | Next Topic
Printer-friendly copy    
Conferences Eyrie Miscellaneous Topic #284
Reading Topic #284, reply 0
Charter Member
19721 posts
Aug-01-14, 10:56 PM (EDT)
Click to EMail Gryphon Click to send private message to Gryphon Click to view user profileClick to add this user to your buddy list  
"Elder Days Story Time: Cablevision Employee"
   So, the disgruntled Cablevision employee.

This is going to take a little explaining, in the form of small geography and technology lessons. Geography first:

The UltraNet main office was in Marlborough, Massachusetts, which is in what IT-industry hipster types called "Metro West" - the office park jungle that sprawls between Boston and Worcester, basically. Here is a Google Maps screenshot of the general area we're talking about.

The orange pin marks the approximate location of UltraNet headquarters.

You will note that Boston has two major highways that form sort of a double bullseye ring around it: I-95 slash MA Route 128, which is just outside the city and runs through exurbs like Burlington, Waltham (where I used to live), and I-495, which is outside that (relative to Boston itself). UltraNet was just on the outside of 495.

Right. On to technology. This was the fall of 1996; the technology for gettin' on the Internet was a little different in those days. Digital subscriber line services were very expensive - at UltraNet we didn't even offer them. Most ordinary people were still getting their service via modem dial-up, and the bulk of our business customers were using basic-rate ISDN (a digital networking specification so balky and tiresome to provision that the common expansion of its initials was "It Still Does Nothing"). We had a handful of particularly-well-heeled residential ISDN customers, and our top few business customers had dedicated lines (T1s, mostly), but for the most part, residential dial-up and small-to-medium-business ISDN were our thing.

Both dial-up and ISDN had distance restrictions - dial-up customers obviously wanted the number they called to be a local call, and ISDN clients had to be within a specific range (I forget what it was) of the originating equipment - so we, like all ISPs of the day, had a network of points-of-presence (POPs) around our operating area. These were little rented spaces where a whole horde of phone lines would come in and be connected to big banks of modems racked next to serial comm servers, which in turn talked through a dedicated line back to our giant Cisco 7513 router at the mothership in Marlborough. (I got into the ISP net-ops gig by way of doing tech support for those comm servers, in fact.)

One of the main things we did on the "hot seat" at UltraNet was monitor the links to the POPs, since obviously, any one that went down would be useless to the customers in its area, and that wouldn't be good. Of particularly critical importance was the POP in Waltham, just inside 128 west of Boston, because that was where virtually all of our business ISDN lines terminated, to be backhauled via our biggest dedicated line back to Marlborough. If we lost that one, we lost... well, basically everybody.

So about 5:30 one rainy afternoon, after most of the day shift had gone home, I was sitting in the ops room doing my thing when an alert popped up on the network monitoring station. While I was looking into that, Tozz (the NOC manager at the time) stuck his head in the door, said he was heading out, and asked if anything was going on.

"Well, Woburn just went down for some reason," I said.

Ordinarily that would be no particular reason for Tozz to stick around - it was the sort of thing staffers like me were there to deal with - but he put down his bag and came in to have a look. A quick run of the diagnostics showed that the problem was the backhaul loop between the Woburn POP and Marlborough, but all we could tell from the tools we had was that it wasn't passing traffic - not why not. I could call the POP from the phone on the ops desk, and a modem would answer, so we knew the power wasn't off (that happened sometimes). The port on the 7513 was fine, so that left the router at the other end or an interruption in the circuit itself.

We didn't own the dedicated circuits we used for backhaul from our POPs; they were leased from a higher-level provider called Teleport. The procedure, then, for when something like that happened was to call up their NOC and say "hey, are you having a problem with circuit $whatever? because it's gone dark," and if they said "nope, it looks fine," we would know to call our on-call plant services guy and have him drive out to Woburn and kick the router.

In this case, the answer was, "Yeah, we see it too. I dunno, we're looking into it."

"Hmm, well, you got this," said Tozz, and he was about to leave, when - beep! - another POP went down. Lexington, this time. Well, shit. I checked the phone again - still power in Lexington - then called Teleport. Same problem as Woburn, which was still down. Tozz went back to his cubicle to let customer service know, because with two POPs down they were going to be getting a lot of calls from annoyed dial-up users (we didn't have to deal with dial-up users directly in the NOC).

Ten minutes later, Waltham went down.

Remember what I said about the Waltham POP a while ago? Yeah. When Waltham went down, everything went down, in terms of our business ISDN customers. Sure, it was after hours, and so not nearly as monumentally painful as it could've been, but still, that was dozens of customer circuits out all at once, and one guy - me - at the other end of the phone number they all had to call to find out what was going on. And I was on the phone with Teleport trying to find out from them what was going on.

At this point Tozz, like a hero, started calling down the customer list to tell them what was happening, so I could focus on that... and a few minutes later, Newton went down. This time the guy at Teleport called me to let me know, "Yeah, we see it, it's the same thing."

Now. Scroll back up for a second and have a look at that map. See the red dots? The big one is Waltham. The two above it are Lexington and Woburn. Newton is below it...

... and 10-15 minutes later, Needham went out.

I didn't have a map handy, so it took me that long to realize what was happening. I called Teleport back and said, "Hey, uh, this is just a shot in the dark, but I'll bet you the next POP we lose is Dedham."

Guess what happened.

"Son of a bitch," said the guy from Teleport. "I have to make some calls."

No more of our POPs dropped, but the ones that had gone weren't coming back, so our network was effectively crippled for most of Metro West. Tozz kept working the phone; so did I, now that there was nothing else I could do troubleshooting-wise. Eventually he finished up and headed home; there wasn't anything else we could do. Luckily, most of our business clients were good about it (admittedly, we did kind of throw Teleport under the bus by telling our customers that it was our own vendor that was having the problem, but that was at least true), and a lot of them were closed, anyway, but I didn't even have an ETR (estimated time to repair) from Teleport on any of the affected circuits, which was troubling.

Then, about 10 o'clock that night, the guy from Teleport called back, sounding weary but triumphant, and said, "You'll never guess what happened to your POPs."

And what happened to our POPs, he reported to me, was this:

A disgruntled former employee of one of Teleport's other customers, Continental Cablevision (later to become MediaOne, now part of Comcast), took a shovel and an axe, got in his car, and started driving the 128 arc from US 1 north of Boston to US 1 south of Boston. Every few miles, he'd pull over, get out of the car, walk out into the woods, dig up a fiber-optic cable, chop it off with the axe, and then walk back to his car. And so every 10 or 15 minutes, another block of Cablevision subscribers would lose their service, which was also being hauled on Teleport dedicated circuits. It had nothing to do with UltraNet; the guy probably had never even heard of us, let alone knew that we backhauled our POPs on Teleport's fiber.

The cops caught him before he could axe the line in Canton, and then Teleport's teams had to go out there and work all night in the pouring rain to fix the six cuts he'd managed to make. By the time I left work that night, Woburn was back up; everything was back to normal when I got in the next afternoon.

And that's the Cablevision story.

Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.

  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top

  Subject     Author     Message Date     ID  
Elder Days Story Time: Cablevision Employee [View All] Gryphonadmin Aug-01-14 TOP
   RE: Elder Days Story Time: Cablevision Employee Nova Floresca Aug-02-14 1
   RE: Elder Days Story Time: Cablevision Employee SpottedKitty Aug-02-14 2
   RE: Elder Days Story Time: Cablevision Employee Proginoskes Aug-02-14 3
   RE: Elder Days Story Time: Cablevision Employee Offsides Aug-08-14 4
      RE: Elder Days Story Time: Cablevision Employee StClair Aug-08-14 5
      RE: Elder Days Story Time: Cablevision Employee Zemyla Aug-10-14 6

Conferences | Topics | Previous Topic | Next Topic

[ YUM ] [ BIG ] [ ??!? ] [ RANT ] [ GNDN ] [ STORE ] [ FORUM ] GOTW ] [ VAULT ]

version 3.3 © 2001
Eyrie Productions, Unlimited
Benjamin D. Hutchins
E P U (Colour)