It wasn't so bad at UltraNet, because we could leave - had to, in fact, there was nothing to eat in the building - and there were a number of decent restaurants nearby. It was also enough time to swing by the supermarket or the OfficeMax and pick up sundries as the situation required, or even, if one was particularly speedy and knew the secret ninja back way, hit the Solomon Pond Mall (if one had a particular item in mind and knew more or less where in the building it was).

Tangentially, thinking of OfficeMax reminded me of my favorite trouble ticket. We had a trouble ticketing system in the NOC that we were supposed to use for anything that we couldn't fix in a minute or two on our own - problems with the network architecture, hardware failures that would require physical repairs, and so on, as well as customer complaints and problems with stuff outside the company's direct control (such as Network Solutions fucking up someone's domain name registration). Issues could be graded with severities from four to zero, with the lower numbers being worse. Severity zero was reserved for things like the 7513's backplane failing (that happened), lightning strikes, Communist invasion - real end-of-the-company's-world type stuff.

One day someone on the third shift accidentally filed a failed disk on the NNTP server as severity zero instead of two (look at your nearest numeric keypad and you can see how that might happen), which caused the trouble ticketing system to page Dwight, the director of network operations, at 2:30 in the morning. Later that week, at the weekly everybody-who's-here NOC meeting, he belabored the point exhaustively that severity zero was for real the-company's-at-stake problems only and that he, his wife, and his wife's old college roommate had really not appreciated the false alarm.

Really. He told us that.

Also: Third shift wasn't at the weekly everybody-who's-here NOC meeting, because third shift. So the guy who did that wasn't even there to be chided.

That Sunday at 10:55, just before clocking off for the night, I filed the following trouble ticket (reconstructed from memory):

NETWORK OPERATIONS TROUBLE TICKET #[large number]
FILED BY: HUTCHINS Benjamin D
DATE: Sun Month XX 23:55 EST 1997
SEVERITY: 0

SUMMARY: NOC Lava Lite(tm) inoperative

DESCRIBE PROBLEM: The Lava Lite(tm) in the Network Operations Center stopped working at approximately 22:45.

DESCRIBE STEPS TAKEN: Power-cycled Lava Lite(tm); no effect. Applied lockout/tagout procedure to Lava Lite(tm)'s electrical power supply. Attempted to remove Lava(tm) canister. Lava(tm) canister still near operating temperature; burned fingers. Said bad word. Protected fingers with paper towels and tried again; operation successful. Removed light bulb. Examined bulb.

CONCLUSION: Open circuit; filament in bulb no longer continuous.

ACTION REQUIRED: Management-level employee to obtain PO and purchase replacement bulb (specification: [the gobbledegook printed on top of the bulb] at suitable retail outlet e.g. OfficeMax. Install replacement bulb per manufacturer's approved procedure. Reverse lockout/tagout and restore electrical power supply to Lava Lite(tm). Allow 1/2 to 1 hr for Lava Lite(tm) to return to operating temperature.

ESCALATED: Sun Month XX 23:57 EST 1997
ESCALATED TO: MANAGER, Dwight

When I got to work the next day at noon, the Lava Lite™ in the ops room was working again and the ticket was marked,

CLOSED: Mon Month XX 08:49 EST 1997
CLOSED BY: MANAGER, Dwight

He never said a word to me about it.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.