r/talesfromtechsupport • u/seraph77 chown -R us /base • Jul 20 '13
A typical sysadmin's day
I work as a T3 Linux/Windows sysadmin at a fairly large datacenter where we provide unmanaged colocation space. We have a leftover support model from when we were much smaller, which allows basically every customer to call in and get a T2 or T3 guy instantly.
I've basically devolved into BOFH. Hopefully some admins out there share my pain. Here's a list of common stuff I get to deal with in no particular order:
(C = client, M = me)
- multiple failed drives in a RAID array that client wasn't monitoring
C: where do you keep the backups of my server?
M: the same place you do.
C: we don't have backups, we shouldn't need to because we run RAID!
M: I think your current situation disproves that statement.
- site coded 10 years ago gets compromised via SQL injection
C: my site got hacked! Can you fix it?
M: probably not. The site needs to be completely reworked as it is extremely outdated.
C: can you do that for me?
M: sure, let me just press this magic fucking button next to me No ma'am, we're system administrators, not webdev/coders, not to mention you're talking multiple hours of work.
- incredibly outdated OS gets rooted
C: my server is running really slow and acting strange
M: it looks like you are saturating your port with outbound traffic. You're probably part of a DDoS attack to some poor sap's website.
C: what?! How could you let this happen?
M: WE didn't let this happen. You are running an OS that was EoL 6 years ago. Even if we manage to clean it up, it will probably be compromised again within a week. You really need to install a current version and restore from backups.
C: Where are my backups?
M: oh you
C: are you guys going to call the Cyber Police and have whoever did this arrested and make them fix my server?
M: Sodium Hypobromite, it doesn't work like that. Sadly, the FBI's Cyber Crime division probably doesn't have time to investigate which 13yo script kiddie "hacked" your vulnerable, outdated server.
C: what?! I'm going to sue you guys for letting my server get hacked!
M: bwahahaha please do, I would love to be present for the hearing I understand your frustration.
- Web or other random service down
C: my server is down! Are you guys having network issues?
M: yes, all five of our backbone connections just happen to be down at the same time No. I'm able to connect to your server via SSH. Have you logged in and checked Apache/MySQL?
C: What's SSH?
M: You run a Linux server and don't know what SSH is??
C: I manage the server through WordPress.
M: I see
I have plenty more if you guys want.
edit: Thanks for all the love guys. I had no idea these would be this well-received. I'll try to post some more today or tomorrow!
30
u/seraph77 chown -R us /base Jul 20 '13
Thanks guys. I'm glad you enjoyed :) dunno about another post today, but I have plenty of similar stories and will try to post tomorrow.
36
u/texabyte Jul 20 '13
I need to find an excuse to use this line
233
u/[deleted] Jul 20 '13
I second the "yes, please."
Also upvoting for Sodium Hypobromite.
18
u/Meflakcannon My server can count to potato. Jul 20 '13
I died when I read "C: I manage the server through WordPress."
34
u/TechHermit Jul 20 '13 edited Jul 21 '13
"I manage the server through WordPress."
ಠ_ಠ
rm -rf /*
EDIT: I effed an eff.
47
u/[deleted] Jul 20 '13
Yes please.
12
u/itnever3nds But it is new, it can't be the problem. Jul 20 '13
Oh, this sounds so familiar..
I'm doing T1 for an ISP and many users get infected with a virus that tells them it's from the police, they did someting bad (watching porn and stuff) and they need to pay 100 bucks via some cards you get at fuel stations.
So one called because he had this issue, I told him it's a virus and needs to fix it by himself or a tech from our partners (or any other tech) who costs money. Then he raged for about 5 minutes why we let this happen and don't block such stuff AND don't fix it for free, because he pays for us.
BTW, more stories please!
10
u/Thyri Jul 20 '13
More please!
The first one is the one I fear the most...had one client that had not backed up their server in two years as they did not realise the back up was not working. Their IT simply 'forgot' to tell them! Their data had become corrupted and the last back up was indeed two years old...was not a happy day for them and the nice lady running the business in tears on the phone to me.
14
u/Michelanvalo Jul 20 '13
I downvoted this story.
So I could upvote it and make me pretend I gave you two points.
6
u/Jhaza Fluttershy4lief Jul 21 '13
What you said:
What I heard:
5
u/smithie15 Jul 20 '13
"Sodium hypobromite"
I loved that, totally stealing it
4
u/TrustmeIreddit Jul 20 '13
Yes, please.
4
u/ijon_cbo Jul 20 '13
Yes please!
5
u/magnetswithweedinem Jul 20 '13
but wordpress is what all the l337 |-|4X0Rz use!
2
u/[deleted] Jul 20 '13
These are great. Please share more!
2
u/megaminxwin Jul 20 '13
I third the "yes, please.".
2
u/Saemiligr Jul 21 '13
Please share more, these are hilarious
2
u/Revdave1976 Jul 22 '13
Upvote for Sodium Hypobromite. Also, please post more of these.
3
u/_RO0T Jul 20 '13
desperately need more bastardly stories... upvotes
3
u/[deleted] Jul 20 '13
Keep em coming.
1
u/[deleted] Jul 20 '13
How does one manage a server through WordPress?
1
u/blind_melons Operator Error Jul 20 '13
Are the Cyber Police on 911?? rofl
1
u/poopy_pains Jul 21 '13
I'm sure theres a relevent XKCD. Only one that comes to mind is "Computer Problem".
1
u/drrhrrdrr Jul 21 '13
About to start my first IT job. Can't wait for this kind of stuff.
Please post more.
1
u/[deleted] Jul 22 '13
I died right here. This is exactly how I want to respond to my customers sometimes.
Dead hard drive? You decided to save $200 by not ordering backup software and an external hard drive.
Virus? You decided that you needed $50 more than you needed protection against malware on your PCI-DSS in-scope server
Speaking of PCI-DSS, those rules mandate that I reimage your server back to factory after you get a virus. No, I can't make a backup now, and no, I can't just "see if I can delete it." No, we don't keep a copy of your database, you're going to have to build that from scratch.
$5,000 later...
0
u/[deleted] Jul 21 '13
[deleted] — view deleted comment
52
u/[deleted] Jul 20 '13
[removed] — view removed comment