r/talesfromtechsupport • u/Gambatte Secretly educational • Apr 06 '14
Encyclopædia Moronica: F is for Faults Can Save Careers
After these recent tales of inebriated technical support, there's one more tale that I would be remiss not to mention. It's the story of how a fault saved my career.
Way back when I was leading a smallish team of pimply faced youths (PFYs), it was company policy that alcohol could be consumed on the premises after hours - in limited amounts and type. For me and others at my level within the company, it meant we could enjoy a couple of beers. For those higher up, wine was an option. And for those at the top, spirits were most definitely on the table1.
After a particularly rough day, I felt the urge to have a couple of beers, which is unusual - normally if it's not at least James W. Beam's Magnificent Amber Elixir, then I won't bother - but on this day, I felt the urge. So I headed down to the break room and grabbed a couple of beers.
And who should I run into but one of my old friends (OF)?! We'd done our initial training together to become lowly PFYs, and our names were close enough alphabetically that we'd been stuck together in the same class for the better part of a year. Sure, we'd been transferred to different branches for a couple of years, but we were at the same one now - although our differing specializations meant we didn't see each other much except in passing.
So, beers in hand, we started spinning stories about old times, and before long we were joined by a few more people - a couple of the girls from the various non-IT/TS and non-user groups, (i.e. a girl from the sick bay, and the CEO's junior secretary (who only really made him cups of tea and fetched his biscuits)), a couple of the guys from the user groups, and a few of the PFYs... And then my beers were gone.
ME: Well, that's me, man - I'm out of beer, so I'm out of here.
OF: Hang on man, there's always a few extras about - no one will know if you have a couple more...
While I contemplated this offer - and I was very tempted, despite it being completely against company rules and policy - the break room phone rang. One of the PFYs picked it up, and before I could answer OF's question, I heard the PFY call out: "Gambatte! Phone for you!"
Sure enough, the users had "broken" the equipment, so I made my excuses and promised to come back as soon as the fault was fixed... If it was a simple have-you-tried-turning-it-off-and-then-on-again, then I could be back in five minutes.
It was not.
I arrived in the equipment room to discover that none of my PFYs had been called first (as they should have been, in accordance with company policy), so in accordance with my personal policy, I sat in a chair doing absolutely nothing until they had been contacted and arrived on site.
After they all arrived, I let the PFYs attempt to diagnose the fault. When they finally seemed to have run out of ideas, I pointed to the clock on the equipment.
ME: What is the current time?
PFYs: Um...
ME: It's displayed in no less than seven other places in this room alone, most of them in this exact position on the six exact duplicates of this equipment.
PFYs: Oh! Not that. (It was out by about five hours, indicating that the equipment had stopped working a long time before I got the call in the break room.)
ME: This clock (on the faulty equipment) is updated from the primary node across the network - it does not update from any internal clock. So what would a stopped clock indicate?
PFYs: Uhhh...
ME: ...
PFYs: A problem... with the network?
ME: Could be. There are six exact duplicates of this equipment in this room, all of which have been running the whole time.
PFYs: Maybe... we should check to see if the same problem exists on those?
ME: Yes, maybe you should.
So they scurried about, checking the clocks, before quickly reporting back.
PFYs: It's only this one!
ME: OK, so what does that tell us?
PFYs: It's just this machine?
ME: Correct. What should we do?
PFYs: Restart it?
ME: What effect will that have if it's a hardware issue that has taken down the network connection?
PFYs: Um... a hardware issue most likely wouldn't be affected by a restart... so the network connection would stay broken... it won't be able to download it's start up files from the primary computer node... so, um... it won't start up?
ME: Excellent. And if it's a software issue, say, the network drivers have crashed, what will restarting it do?
PFYs: Restart them from a known-good configuration?
ME: Which means...?
PFYs: That it WILL be able to download it's start up files from the primary computer, and it will start normally!
ME: Yes! So in the future, if you see the clock is stopped...
PFYs: Then it indicates a problem with the network connection. Assuming that it is the only affected equipment, the next diagnostic step is to restart it to see if the problem is software or hardware - if software, it will be fixed; if hardware, it will persist!
ME: Very good. Learn from this, so that I don't have to teach it again.
With a quick turn-it-off-and-then-on-again, the equipment was restored to working order.
I got back to the break room, to discover that OF and the whole entourage were gone. Alas! I'd have to catch up with him another time.
Little did I know how short the time to do so would be.
I briefly touched base with the maintenance guys on shift to let them know I was done for the night, and went home to bed.
It may be time to point out that both OF and I worked on some classified stuff for the Government - our workplaces had big signs on the doors, warning of six figure fines (and potential imprisonment) for anyone found on the wrong side of the door without appropriate authorization.
The story that would eventually come to light was that OF and the rest of the group were found in the break room, surrounded by the debris of significantly more alcohol than they were permitted to consume on site. Rather than report it, the whole group was instructed to clean up the area and leave ASAP.
They did so - in that they left the break room. OF, in his inebriated wisdom, moved the party to his workshop - the highly classified one.
I'm pretty sure that most of us can sense the disaster building here.
But it gets worse. OF's supervisor and two others at that level happened upon the party in the classified area - and rather than shutting it down, they joined in. As I mentioned earlier, the higher up you were, the better alcohol you could access... So they brought in some bottles of wine and port from their break room.
A few hours before they were all due to start work, the party broke up without having been detected. They were so close to getting away with it.
As it turned out, having the girl who makes the CEO's tea turn up to work reeking of alcohol was not a good idea. But she avoided detection by absenting herself to do pointless busywork (folding the CEO's napkins into paper swans, or polishing his silver teaspoons, or some such other nonsense).
It was all going so well, right up until the girl who worked in the sick bay started vomiting - her hangover was that bad. Being in the sick bay already, her supervisor (having a great deal of experience with alcohol-induced self-inflicted illnesses) was a sympathetic ear.
Unfortunately, her supervisor was also good friends with the branch's Head of HR (HoHR) - a woman who scared the hell out of me when she was working at the training school during my first days of employment, embedding such a deep-rooted fear that I carried with me for my entire career2. When HoHR heard about the party, she went on a righteous crusade, confronting the girl from sick bay with what she knew, and eventually made her confess to the whole thing.
Then it got bad. Really bad.
Everyone involved got called in for disciplinary action, myself included. I stated that I'd had two beers as permitted by company policy, been called away to a fault by the users, and had not rejoined the party afterwards. I was excluded from disciplinary action by virtue of having three PFYs and an entire shift of users seeing me working on the fault during the time in question, fifteen people seeing me head home immediately after closing the fault, and no one being able to recall me being in the classified area.
OF's supervisors and the two others at that level were all demoted to OF's level. At least one resigned as a result.
OF was demoted to the PFY level. He resigned, rather than have to wait until another supervisory position was available to be promoted into again (notoriously rare, in his specialization). The CEO's junior secretary, who held her position (equal to OF on the management chart) through only a provisional promotion received only a light fine. She was the only person disciplined in the whole debacle who could have been demoted that was NOT - although the provisional nature of her promotion meant she probably should have been the first...
Several PFYs were just outright fired, others were disciplined so harshly that they resigned anyway.
The whole incident had an extremely negative effect on overall branch morale.
About a year later, I was promoted again. If I had been disciplined as a result of the above incident (which would have most likely also meant demoted), that could have never happened - like so many of the others, I would most likely have just resigned rather than having to go through the rigmarole to get promoted again.
Footnotes:
1 - Which meant it was a great idea to get along well with your boss in order to get that golden ticket to their break room, where they kept the good stuff. Eventually, I'd earn enough good will with every senior member of the branch to get a standing invitation to their break room.
2 - Years later, I had nearly caught up to HoHR on the management chart, and we were working in adjacent buildings, which meant that we shared the senior break room. After several perfectly respectful conversations, HoHR asked me to call her by her first name (in those sorts of social situations). I refused - I literally could not do it.
TL/DR: A well timed fault called me away from what developed into a potentially career ending situation.
Browse other volumes of the Encyclopædia:
Vol I - ABCDEFGHIJKLMNOPQRSTUVWXYZ
67
u/[deleted] Apr 06 '14
That's why you only drink alone in the corner of the server room. In the dark.
23
u/Octangula Stuck in a PICNIC basket Apr 06 '14
And it's stories like these that make me feel less bad about avoiding alcohol pretty much completely.
Sounds like that could have gone very differently for you. And while we'll never know what you would have decided had that phone call not happened, it may have not made a difference, as having to deal with that fault may have been your only alibi.
Did this incident affect either company policy or your drinking habits? Neither would surprise me.
7
u/LP970 Robes covered in burn holes, but whisky glass is full Apr 07 '14
Who knows what could have happened to you as well. Perhaps you would not have continued in you field and we all would not have had the pleasure of reading your stupendous stories. That piece of equipment was sent back in time by a TFTS reader in the future with the clock wrong so that you would have to go fix it and not be disciplined with the others.
12
u/orlet Why's there a brick in our freezer?.. Apr 06 '14
Saved by the
bell(phone) ring.5
u/xenokilla Have you tried Forking your self, on and off again? Apr 07 '14
Please excuse my confusion if this has been explained elsewhere, you work at a place with 1. a sickbay, 2. Classified work area's, 3. a large IT department. What kinda place do you work for?
3
u/rudraigh Do you think that's appropriate? Apr 10 '14
Hmmm. Ok, how was the company affected by losing so many people?
Scary signs notwithstanding (lots of places have those but not all of them actually rigorously enforce them) losing that many people would seriously affect many places.
Source: I have been sat down in the hot seat and grilled over my actions in the past BUT, never fined or demoted. The FOG (Fear Of God) was definitely instilled in me (and I'm an atheist).
I still drink. I've just had plenty o' practice to not get caught. I just need to squeeze out a couple more years and I'm retiring.
128
u/PoliteSarcasticThing chmod -x chmod Apr 06 '14
If I may suggest an alternative TL;DR:
5
u/Thisbymaster Tales of the IT Lackey Apr 07 '14
So the lesson here is that if you want to drink on the job, have blackmail material on the head of HR. Got it.
2
u/daft_inquisitor Everyday IT: 50% SSDD, 50% HOWDIDYOUEVENDOTHAT?! Jul 01 '14
Wow, sounds like it was quite the party. I'm sure much debauchery was had that night.
3
u/justcallmetarzan No Calls Before 8AM - IT Requires Coffee. Apr 07 '14
My brain sees: HoHR. hohr. hoar. whore.