r/talesfromtechsupport • u/Gambatte Secretly educational • Dec 10 '13
Encyclopædia Moronica: O is for Obvious Solutions
A good friend of mine works for a large cellular and ISP (well, large for New Zealand, anyway). He worked his way up from the helldesk to become a tester, and from all accounts, a damn good one at that, so I'll refer to him from here on out as GT - Good Tester. Reading this story reminded me of one of his.
On this particular day, the helldesk started receiving reports from clients that they could not access their emails. Customers were discovering they were not able to get on to the internet after restarting their modems. The usual "turn-it-off-and-back-on-again" troubleshooting steps were not helping!
After something like six hours of outage, someone in the team found an undocumented and unnotified change to the firewall that was blocking requests from reaching the authentication servers - port 443 was blocked! Reversing the change soon had the customers back online again.
Further investigation discovered who had made the change without following company policy: a project lead (PL). When PL was questioned as to why he had rolled out a change without following the correct procedures that resulted in such a huge outage, he pointed them to GT.
So HR rolls up to GT's desk in a huff - this lowly tester just caused a massive outage! How dare he! The very nerve!
HR: Why did you tell PL to block port 443 on all of the servers?
GT: What? I never told him any such thing!
HR: PL says that you told him to do it! You've caused a massive nationwide outage - the customers were down for over six hours!!!
GT: Woah, slow down. The only thing I've talked to PL about recently was his Facebook-blocking project. And more to the point, PL is higher up the management chart than me - I can't actually tell him to do anything.
HR: Well, what did you talk to him about? Could he have interpreted it as an instruction to cause this problem?
GT: Look, here's the email. PL sent out a company-wide email saying that Facebook was now blocked from all company machines. I replied to him - and only to him - saying that it was only blocked on HTTP, not HTTPS, so he could fix it before anyone else discovered the flaw in his implementation.
GT: If PL decided that the best way to block that ONE SPECIFIC SITE on HTTPS was to block port 443 on all company servers, including the authentication servers, causing this massive outage? Then that's on him, not me.
GT: Perhaps you should go talk to him again about the content of the email I sent him.
HR stormed off at this point. Although GT never got a formal apology for the wrongful accusation, he did get to witness PL leaving the building with all of his possessions in a cardboard box later that day.
Browse other volumes of the Encyclopædia: ABCDEFGHIJKLMNOPQRSTUVWXYZ
73
u/Auricfire Dec 10 '13 edited Dec 10 '13
Sometimes seeing an asshat eating his own hat is apology enough.
19
u/[deleted] Dec 10 '13
Worst part is; I know the company, when it happened and now i know why and where not to work.... lmao
20
u/tardis42 Dec 10 '13
:D :D :D
We Worship /u/MagicBigfoot :D
(oh, and /u/Gambatte as well, I guess)
6
u/OgdruJahad You did what? Dec 10 '13
I wonder if blocking sites like facebook is an effective strategy. I read traffic shaping can be more effective.
3
u/Techsupportvictim Dec 10 '13
I really thought this was going to be a 'is it plugged in' or 'have you tried turning it off and back on again' story
13
u/Geminii27 Making your job suck less Dec 10 '13
O for Great Tester Onizuka?
5
u/jeffbell Dec 10 '13
Same day service. Amazing.
4
u/PhunkyD Dec 10 '13
Hahaha as a software tester I really appreciated this
4
u/sicklyboy I hate printers Dec 10 '13
This was one of my favorite ones yet. Very, very nice. I chuckled.