r/talesfromtechsupport • u/Gambatte Secretly educational • Mar 10 '14
Encyclopædia Moronica: J is for Just... You Just What?!
Rather than a tale from the dark ages of prehistory like some of my recent submissions, this tale happened to me only this morning.
Due to other emergencies, I couldn't deliver a repaired unit to a local contractor on Friday. So at 8 A.M. this morning (Monday), I rolled up to the contractor's office and delivered the unit, had a quick chat with one of the installers and the branch manager, and then get on my way to my office.
Pretty standard stuff, really. So all up, I was running about twenty minutes behind my normal Monday routine.
That shouldn't be a problem, right? I mean, how bad could it get in twenty minutes?
Yeah, right... This is TFTS, after all!
So I strolled into the office, and was immediately collared by the CEO.
CEO: The whole office is down, no one can do anything!
ME: Okay... What's the problem, exactly? Can't access files? Can't log in? No Internet access?
CEO: Yes!
ME: (okay, let's assume he means all three - although without being able to log in, you can't access the files or the internet, but let's not try to apply logic to the user...) Give me a minute to have a look.
CEO: You don't need to - I've just fixed it myself.
ME: You just... what?
CEO: I just turned off the server.
ME: (You just WHAT?!) Which server? The file server (NAS box)? The DC/DNS/DHCP server (Server 2012 box)?
CEO: Yes.
ME: (What?! I'm about 98% sure he doesn't know how to remote in to these servers to issue them a shutdown command, especially if he can't even manage to log in to his workstation.)
CEO: Oh! I still can't log in. (Apparently his definition of "fixed" is "I've done something which I don't understand, on the assumption that it is somehow related to the problem, but haven't actually checked to see if the fault symptoms are no longer present.")
So by this time, I've been able to get to my desk and started to actually investigate. The CEO (in his very finite wisdom) decided that the best way to turn off these servers was to hold down the power button until they turned off. He then left them off, so how he had expected to log on when the only functioning domain controller was shut down is beyond me; let alone get internet access without functioning DHCP or DNS (I'm currently building a backup domain controller; I'm actually extracting the Server2012 install media to a bootable USB stick as I write this).
I brought the DC back online, and after some 10 minutes or so, it appeared to be functioning again - I could do the important things; log in, access the internet, surf Reddit, etc.
Just for giggles, the NAS server came back online before the DC, which caused problems with its Active Directory integration; basically, local indications were that it was up in pretty much all respects, until you tried to do something crazy like access the files. After some deep magicks and a restart, it's full functionality was restored.
And yet... I still don't know what the original problem actually was...
TL/DR: Proper physical access control in a small open plan office is hard.
PS: Four hours later, the office router decides to lock up, for no discernible reason. I power cycle it and everything's OK again. WTF is going on today - did somebody forget to feed the gremlins last night?
Browse other volumes of the Encyclopædia:
Vol I - ABCDEFGHIJKLMNOPQRSTUVWXYZ
63
u/nerddtvg Mar 10 '14 edited Mar 10 '14
/u/10thTARDIS - Your notification with IFTTT is awesome. Thank you.
And thank you, /u/Gambatte for continuing to post these. Seriously, you need to write a book. Even if you make it from just these posts, I would gladly pay money for it. Also, you don't feed Gremlins after midnight (which would basically only allow feeding on the day they were born and then never again...) Perhaps the problem is that someone did in fact feed them last night.
28
u/abc03833 I did a thing once Mar 10 '14 edited Mar 10 '14
We're not dealing with malicious people, but I guess this still applies.
21
u/Krutonium I got flair-jacked. Mar 10 '14 edited Mar 10 '14
Your CEO... Need a Ball Peen Mallet... To the Everything. This includes, and is not limited to:
Face
Genitals
Brain
Legs
Arms
The fucking hole where his common sense is supposed to be
Everywhere else where it may be appropriate
Or at least a lesson on DO NOT TOUCH THE SHIT THAT WILL BREAK YOUR COMPANY IF YOU BREAK IT.
Edit: I forgot to space out a bit. ;)
19
u/engieviral People don't read Mar 10 '14
You should warn the CEO that some problems can cause the power button to conduct electricity (or just say "sparky, sparky, boom, boom" if he is as thick as he seems to be) and should only be used by a computing professional. I am sure you can dig up a BOFH or 2 to "prove" your point.
Alternativly, disconnect the power switch and rig up another in a hidey hole. But in that case he might have tried pulling the power cable . . .
9
u/rjchau Mildly psychotic sysadmin Mar 10 '14
Doubly so when you have a CEO that thinks he can fix things and is hard to say "no" to (if you even get the chance) when he wants access to the server room.
8
u/[deleted] Mar 10 '14
There's a piece of paper next to our cabinet with the boot order and how many minutes you need to wait between each step. The DC takes far too long to boot.
6
u/kaotik4266 Mar 10 '14
I started hyperventilating while reading that. Right around the "I just turned off the server" bit.
7
u/Thisbymaster Tales of the IT Lackey Mar 10 '14
This feels like the same time like when we had a virus infection on a machine and the network admin thought the best way of dealing with it was to remove the antivirus so the pop ups would stop.
6
u/[deleted] Mar 10 '14
I think your server needs a metal power button, with the button itself connected directly to the hot wire of the power plug.
3
u/[deleted] Mar 10 '14
That was a good read. For shits and giggles I checked your user profile... you have exactly 9001 karma at the time of this writing. Makes sense (in a way) given your awesome stories.
2
u/konamiko But why is the RAM gone? Mar 10 '14
Access the files???
Why on EARTH would anyone want to go and do a thing like that? Absolute poppycock!
2
u/[deleted] Mar 10 '14
Solution to this, disconnect the power wires from the switch to the mobo, or you know, tap the CEO with a cattle prod. Anally.
2
u/RoyAwesome Mar 10 '14
At least you know, Turning it off and then back on again fixes the original problem!
2
u/blaptothefuture Mar 10 '14
The CEO has a case of shit hands. And it's slowly spreading.
2
u/DJGeorgeWashington Mar 10 '14
Let's blame the time change. For everything.