Go back to previous page
Forum URL: http://www.eyrie-productions.com/Forum/dcboard.cgi
Forum Name: Eyrie Miscellaneous
Topic ID: 283
#0, Elder Days Story Time: The Customer Database
Posted by Gryphon on Jul-30-14 at 07:18 PM
I worked the night shift in an ISP network operations center once, many years ago. The shift rotation was set up so that I worked 10-hour days, four on, three off, from noon to 11 PM1. Being in during the afternoons meant that my days were broken up into two phases. From noon to five, I was a hostmaster, which meant I'd sit in my office processing domain name registrations for business customers and managing the company's DNS servers. The shift overlap was also when the management would schedule Important Meetings, so as to get the maximum number of NOC people in on them at once.

At five, when the day shift went home, I would go into the operations room (which was like a miniature version of the Mission Control-style room bigger ISPs have) and do my five hours "on the hot seat", as the network operations engineer2 on duty. This was a lot like being on monitor duty in the Justice League, except with a far smaller chance that the Black Canary would swing by the ops room with dim sum to share. In lieu of that, I would while away the hours doing more of my hostmastery work in between infrequent phone calls from business customers (not a lot of that happening after 5 PM), while keeping an eye on a couple of computers that were set up to report problems with the network as they arose, and boy howdy, sometimes they did. Remind me to tell you about the disgruntled Cablevision employee sometime.

Anyway, another side effect of being in the office during the afternoon was that, even though I worked the night shift, I was there for about half of the rest of the company's regular business day, so I had a feel for what was going on in the joint. For several months, we had consultants in to work on the database system the sales department used to keep track of all the accounts and stuff we had. Every now and then, they would have to take the server down for various reasons, and when that happened, the overhead speakers throughout the office would go,

BEEEEEEP Please exit the customer database.

Customarily what would happen at this point was that the people using the customer database would not exit it. Understand, the overhead message was completely unmissable; between the attention tone and the volume of the PA system itself, it was incredibly obtrusive. Plus, the people using the customer database were getting dialog pop-ups on their screens saying the same thing. They just had no sense of urgency about finishing what they were doing and getting out.

So about five minutes later, the overhead would go,


BEEEEEEP Please exit the customer database.

with just a hint of asperity.

Most of the sales people would take the hint at that point and get out... but the manager of sales was one of those guys who think rules and procedures don't apply to him, because he's the manager, goddammit, so if he was using the customer database at the time, he'd just carry right on with it, figuring if the DB monkeys wanted to mess around with the system, they could just wait until he was finished with his much more important business and was damn well good and ready to let them.

And so, after another five to ten minutes, would come a third PA announcement:

BEEEEEEP Look. Get out of the customer database... FRANK.

Usually that would do the trick, but occasionally Frank would dig in his heels and refuse to budge... at which point the DBAs would play their final card:

BEEEEEEP server01 is going down for a reboot. I hope everyone's out of the customer database.

And then there would be a muffled "Fuck!" from the corner where Frank's office was.

--G.
1 I know that's 11 hours, but we didn't get paid to eat lunch.
2 As you can guess from the subsequent description of the job, I was in no way an engineer. As the son of a licensed professional engineer, I've always found it mildly offensive that the high-tech industry calls virtually everybody who works in any technical capacity at all an "engineer". At Xylogics, for instance, my job title was "Applications Engineer"... I was level 1 tech support. My job consisted of reading the bits of the manual pertaining to what the customers were trying to do and telling them to do what it said, then escalating the ticket to someone who knew something if that didn't work (which it usually did, because the customer hadn't bothered looking in the manual). The only "engineering" going on there was very occasionally social, in the form of finding new and creative ways of not telling the customers a reasonably bright dog could've done what they'd just had so much trouble with.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


#1, RE: Elder Days Story Time: The Customer Database
Posted by Proginoskes on Jul-31-14 at 01:29 PM
In response to message #0
I always enjoy Elder Days Story Time. Oh, Frank.

Xylogics seriously called you an "Applications Engineer"? I am boggled. I mean, it's fairly easy to insert a gratuitous "Engineer" into a job title without making it outright false. I know I've seen job postings for "Technical Support Engineer", which is fairly nonsensical but at least descriptive. I would expect an "Applications Engineer" to be a low-level code-monkey (or, if I was being optimistic, someone with real Systems Analysis & Development experience).

I think the Info Tech HR language has shifted away from abusing "engineer" to abusing "consultant", "analyst", and "developer". Also "experience". Tech support positions are styled "customer experience consultant". If your job is Quality Control, testing for bugs and providing feedback on the user interface, you are a "user experience analyst". And of course every code-monkey in the company is a "developer".

I look forward to hearing about the disgruntled Cablevision employee.


#2, RE: Elder Days Story Time: The Customer Database
Posted by Pasha on Jul-31-14 at 07:22 PM
In response to message #1
A lot of that is for legal reasons. If your job title has "engineer" in it, you're am exempt employee, so they can fuck you just a little more, for instance I don't get paid an increasing OT rate, which blows balls.

--
-Pasha
"Don't change the subject"
"Too slow, already did."


#3, RE: Elder Days Story Time: The Customer Database
Posted by Proginoskes on Aug-01-14 at 11:08 AM
In response to message #2
Really? That's kind of disgusting. I don't believe such a rule exists in British Columbia, so that would be why it doesn't get abused in the same way. The HR dickery I've seen is setting you up as a contractor, so by the legalese of your employment technically the company is your client, so you're not their employee, so they don't have to provide benefits or pay into CPP.

#4, RE: Elder Days Story Time: The Customer Database
Posted by Pasha on Aug-01-14 at 12:49 PM
In response to message #3
>Really? That's kind of disgusting. I don't believe such a rule exists
>in British Columbia, so that would be why it doesn't get abused in the
>same way. The HR dickery I've seen is setting you up as a
>contractor, so by the legalese of your employment technically
>the company is your client, so you're not their employee, so they
>don't have to provide benefits or pay into CPP.

Oh, there's plenty of that, too. Heck, Microsoft actually CHARGES some of it's 'vendors' a 75/mo 'seat fee' to work there, which is just impressive.

--
-Pasha
"Don't change the subject"
"Too slow, already did."


#5, RE: Elder Days Story Time: The Customer Database
Posted by drakensis on Aug-02-14 at 02:39 AM
In response to message #0
LAST EDITED ON Aug-02-14 AT 02:40 AM (EDT)
 
>1 I know that's 11
>hours, but we didn't get paid to eat lunch.

The exact reason for my current daily shift length. If I worked even 1 minute more in a day I'd be _required_ to add in an unpaid 1 hour lunch break to my daily schedule. And as I've pointed out to my managers, I'm not so fond of the building to want to arse around there for an extra hour without getting paid for it, even if I don't need to work.


#6, RE: Elder Days Story Time: The Customer Database
Posted by Gryphon on Aug-02-14 at 03:02 AM
In response to message #5
LAST EDITED ON Aug-02-14 AT 03:03 AM (EDT)
 
>The exact reason for my current daily shift length. If I worked even 1
>minute more in a day I'd be _required_ to add in an unpaid 1 hour
>lunch break to my daily schedule. And as I've pointed out to my
>managers, I'm not so fond of the building to want to arse around there
>for an extra hour without getting paid for it, even if I don't need to
>work.

It wasn't so bad at UltraNet, because we could leave - had to, in fact, there was nothing to eat in the building - and there were a number of decent restaurants nearby. It was also enough time to swing by the supermarket or the OfficeMax and pick up sundries as the situation required, or even, if one was particularly speedy and knew the secret ninja back way, hit the Solomon Pond Mall (if one had a particular item in mind and knew more or less where in the building it was).

Tangentially, thinking of OfficeMax reminded me of my favorite trouble ticket. We had a trouble ticketing system in the NOC that we were supposed to use for anything that we couldn't fix in a minute or two on our own - problems with the network architecture, hardware failures that would require physical repairs, and so on, as well as customer complaints and problems with stuff outside the company's direct control (such as Network Solutions fucking up someone's domain name registration). Issues could be graded with severities from four to zero, with the lower numbers being worse. Severity zero was reserved for things like the 7513's backplane failing (that happened), lightning strikes, Communist invasion - real end-of-the-company's-world type stuff.

One day someone on the third shift accidentally filed a failed disk on the NNTP server as severity zero instead of two (look at your nearest numeric keypad and you can see how that might happen), which caused the trouble ticketing system to page Dwight, the director of network operations, at 2:30 in the morning. Later that week, at the weekly everybody-who's-here NOC meeting, he belabored the point exhaustively that severity zero was for real the-company's-at-stake problems only and that he, his wife, and his wife's old college roommate had really not appreciated the false alarm.

Really. He told us that.

Also: Third shift wasn't at the weekly everybody-who's-here NOC meeting, because third shift. So the guy who did that wasn't even there to be chided.

That Sunday at 10:55, just before clocking off for the night, I filed the following trouble ticket (reconstructed from memory):

NETWORK OPERATIONS TROUBLE TICKET #[large number]
FILED BY: HUTCHINS Benjamin D
DATE: Sun Month XX 23:55 EST 1997
SEVERITY: 0

SUMMARY: NOC Lava Lite(tm) inoperative

DESCRIBE PROBLEM: The Lava Lite(tm) in the Network Operations Center stopped working at approximately 22:45.

DESCRIBE STEPS TAKEN: Power-cycled Lava Lite(tm); no effect. Applied lockout/tagout procedure to Lava Lite(tm)'s electrical power supply. Attempted to remove Lava(tm) canister. Lava(tm) canister still near operating temperature; burned fingers. Said bad word. Protected fingers with paper towels and tried again; operation successful. Removed light bulb. Examined bulb.

CONCLUSION: Open circuit; filament in bulb no longer continuous.

ACTION REQUIRED: Management-level employee to obtain PO and purchase replacement bulb (specification: [the gobbledegook printed on top of the bulb] at suitable retail outlet e.g. OfficeMax. Install replacement bulb per manufacturer's approved procedure. Reverse lockout/tagout and restore electrical power supply to Lava Lite(tm). Allow 1/2 to 1 hr for Lava Lite(tm) to return to operating temperature.

ESCALATED: Sun Month XX 23:57 EST 1997
ESCALATED TO: MANAGER, Dwight

When I got to work the next day at noon, the Lava Liteā„¢ in the ops room was working again and the ticket was marked,

CLOSED: Mon Month XX 08:49 EST 1997
CLOSED BY: MANAGER, Dwight

He never said a word to me about it.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.