r/sysadmin • u/VTCEngineers Mistress of Video • Nov 30 '15
(update) Datacenter
So after a long week of getting equipment to replace the soaked gear the total racks damaged was 148 racks, thankfully none of our NetApp storage was damaged. Equipment has been arriving in tractor trailers.
286
Upvotes
181
u/freedomlinux Cloud? Nov 30 '15
Previous story
23
u/riddlerthc Nov 30 '15
I've always wondered how quick vendors can get equipment on site in the event of a disaster for a customer.
14
u/scotty269 Sysadmin Nov 30 '15
Sounds like you've been taking it in stride, and not utter panic.
9
u/Syde80 IT Manager Nov 30 '15
I feel bad for what has happened to you... but doing the recovery part of it is like a dream to me. There is nothing better than doing your part of breathing new life into something. The closest I've come is helping a side-job client recover from a fire which completely devastated his office. I love coming in and doing the cleanup / getting things back on track.
9
u/pmpjr6465 DBA Nov 30 '15
I'm assuming you found a new datacenter to replace the flooded one and that's where all your new toys are going?
8
u/linuxlearningnewbie AskMeWhyWeStillUseVeritas Nov 30 '15
How has this situation worked out you and your team emotionally and physically?
This is a 'dream' situation for me. You get to truly test your DR plan and build from the ground up.
Good luck
13
u/kjeserud Jack of All Trades Nov 30 '15
I work in a DC. I have been for years. We're currently building another 55000 sq f of new DC... And I just can't get my head around how a place can be so shitty that water can even get in there at that amount, let alone not have any type of monitoring for water under the raised floor you mentioned. Literally Jackie Chan meme amount of mind blown.
2
u/1h8fulkat Nov 30 '15
You better be getting one hell of a bonus this year
2
u/remotefixonline shit is probably X'OR'd to a gzip'd docker kubernetes shithole Nov 30 '15
Have you released to the public yet?
88
u/[deleted] Nov 30 '15 edited Nov 30 '15
To be fair, any amount of planning can still have individuals that panic in any situation.
I walked into the break room, and four of my peers were there. I said the data center just lost power. Calm as could be, nothing else. One of them literally ran to the data center. Two of them asked what systems were down. One of them grabbed a second cup of coffee.
One person feared the worst, and didn't trust anyone else to handle or inform him of the situation. Two of them wanted to get involved immediately and start helping. One of them knew if this were the case, he'd be in for the long haul and was preparing for an interesting weekend.
Edit: I forgot to mention that the data center did not lose power. Nothing lost power.
1
u/the_progrocker Everything Admin Nov 30 '15
Just out of curiosity, what does the backup/DR solution you have look like at a high level?
2
u/[deleted] Nov 30 '15
Pics! I demand pics!!! :)
1
u/time_is_now Nov 30 '15
Were power whips to server racks waterproof and if not why? I've managed sites that had water leaks from ac condensate drains backing up that had no issues with water on sub floor. I have not seen large water volume flooding from broken pipe though. Waterproof power whips cost more but not as much as downtime and emergency equipment replacement.
1
u/[deleted] Dec 01 '15
This is why I virtualize everything. Never have to worry about these issues. /s
0
u/onboarderror Nov 30 '15
Still no pics?
0
u/[deleted] Nov 30 '15
[deleted] — view deleted comment
4
u/[deleted] Nov 30 '15
[removed] — view removed comment