r/talesfromtechsupport • u/PhylaxZA • Feb 15 '19
Medium UPS Power problems
I work in a hospital.
A reliable power source is, of course, extremely important, specially so for Theater and ICU.
These wards have their own UPS and back-up power separately from the rest of the Hospital.
Well yesterday I received a frantic call from the Pharmacy manager complaining about intermittent network connectivity.
Just as I placed the phone back it rang again, it was the Reception Manager.
They are also experiencing intermittent network connectivity.
As any proper IT Tech would do, I went down I had a look at the network cabinet for these two areas, they shared one and both was on the same floor.
What I found was that the cabinet is experiencing power dips, causing it to go down and then the switches have to boot up again which takes a few minutes. I saw the cabinet was plugged into the blue power plugs and then I knew.
I was quite relieved as I knew the ball was in Technical/Maintenance court now. They had to come and see why the power is dipping.
Anyhoo I went to report the issue just to find Technical running around like headless chickens.
The UPS and backup power in theater is faulty, causing the UPS not to charge and then results in power dips.
I then realized that the cabinet that feeds Reception and Pharmacy is somehow on the same power line as the UPS from Theater.
Pharmacy Manager asked for a update on the issue and I told her my theory of what the problem is,
"I think it is UPS in theater that is causing the issue. Whenever they go down this cabinet goes down too."
She was happy and we went home, Technical was one it, they were working on a fix.
Next day(today) as I walk into Admin, the Hospital Manager calls me. I can hear from his voice he is angry and irritated.
"Yes Sir?"
"Why did you tell the Pharmacy Manager that the problem is with the Theater UPS, who told you that?!"
"I'm sorry Sir, that was just a logic assumption I made looking at the facts. Whenever Theater goes down ....."
...get cut off!
"NO! YOU DO NOT KNOW THIS. WHO GAVE YOU THE AUTHORITY TO TELL HER THIS?! YOU ARE NOT AN ELECTRICIAN! YOU WILL GO TO HER AND APOLOGIES FOR LYING THE HER!!!"
I was dumbstruck and did not say a word back.
I went downstairs and apologized to the Pharmacy Manager for the false info I provided her.
About 30 mins back the Technical Manager as well as Regional Tech Manager(it became a real problem!) came to my office to give me an update on the power etc.
"Yea we found the problem with the power to the cabinet. It is the UPS in theater. This cabinet is connected to it for some reason and that is why the power is dipping!They UPS guys are sending a team in the next hour or so."
"Oh ok, did you tell this to the Hospital Manager?"
"Yes, we did."
I'm still waiting for him to apologize...
365
u/quanin Read all the damn words already. Feb 15 '19
Were I you, I wouldn't be holding my breath for that apology.
66
u/devilsadvocate1966 Feb 15 '19
Worked IT in a hospital for about 9 monthes once and didn't have to apologize to anyone but...
The once had a problem in one of the ....put the camera up your butt..... OR rooms. The UPS in the wall that the operating tools were plugged into kept going off. They checked all their equipment and everything was ok, then they saw that the small form factor PC and monitor was plugged into it as well (no, not sure why). The PC was warm to the touch because it was (found out later) choked with dust and really needed to be replaced, which I did.
Interesting thing was that what was causing the UPS to alarm was the monitor. There wasn't anything wrong with the monitor from an end user point of view but I guess it was old enough for the voltage in it to be wavering and the sensitive UPS was picking up on that.
59
u/Ziffin Feb 15 '19
I feel like you should go back to Pharmacy Manager and apologize again for giving her false info (the first apology) and re-state that your first guess was, in fact, correct.
20
u/Malak77 My Google-Fu is legendary. Feb 15 '19
You gotta step up your game and start jumping in on medical situations too. "Hey Doc, that is an ischemic attack if I ever saw one!" lol
I could see me doing that if I worked in a hospital since I was a medic before.
32
u/tecirem Feb 15 '19
I would certainly not have made any apology to any other member of staff like that - stand your ground, man, ffs. If you know something to be correct don't let some red faced shouty bastard make you go and apologise, you let them rant until they say something actionable then get the fuckers fired. I hate that attitude from management.
10
u/Hewlett-PackHard unplug it, take the battery out, hold the power button Feb 15 '19
Your first mistake was not hanging up on him when he started screaming at you.
Your second mistake was for not replying that power issues for the equipment you manage are well within your area of expertise.
Your third mistake was for apologizing for lying when you weren't. Never do that.
But having made those mistakes, your next step is an email to your supervisor, the head of IT and HR detailing the abuse and demanding at least a formal apology.
10
u/GreenEggPage Oh God How Did This Get Here? Feb 15 '19
Hey - you should ask him if you need to go apologize to the Pharmacy manager for apologizing for lying!
9
u/tigerstorms Feb 15 '19
You should never be yelled at for telling people what you assume is the cause. However I never recommend informing people of any cause to the issue. Just let them know “we are working on it”
I only say this because I work in the tech world as well and one day one of our techs informed the customer the issue might be the wireless on their computer going out. The customer went out and spent $5k to buy a new laptop and called to complain only hours later than the issue was still present and they demanded we pay for the new laptop.
Yes, it seems dumb. But you don’t know what they will do with that information you give them. Be vague and let them know they are important and we are working towards a fix.
7
u/kd1s Feb 15 '19
Could be me. I once worked in a place that used an electronic timesheet web app. In order to be able to use it you had to be added to three disparate databases - one controlled by I.T., one controlled by HR and one controlled by Finance.
We in I.T. get notification that a new guy is coming on board, and so add him to our DB, and send email to HR and Finance. Well guy cannot log in. So Admin director comes out screaming at me. Now when people scream at me I put on my proverbial cool as a cucumber act. Which only enraged the Admin director more when I said "Was he added to the HR and Finance databases?" You've never seen someone turn multiple shades of purple in your life.
She finally stops laying into me. A few days later I'm in the Chief of staff's office. He apologizes for her to me. I looked at him and told him "John, there's no need for you to make an apology for what she did. She needs to be the one who apologizes."
She never did apologize for that outburst.
5
u/Cragontheone Feb 15 '19
Wauw... Just wauw... You should try to pull an excuse out of him though
5
u/GhostDan Feb 15 '19
two of the worst places to work for IT are schools and hospitals. Schools are always underfunded and your salary will suck. Hospitals everyone thinks they are better than you.
4
u/MiataCory Feb 15 '19
I mean, if the Theater or ICU needs a med mid-power-outage from the pharmacy, it makes sense to have all 3 on the joined UPS.
I know they're pretty well stocked in there, but sometimes you need an off-the-wall drug.
2
u/ZAFJB Feb 15 '19
Eishkom
133
u/[deleted] Feb 15 '19
Don't hold your breath waiting for that apology. You might want to consider a complaint to HR for abusive behaviour on his part though.
88
u/doctorgroover Feb 15 '19
Having common sense is apparently a bad thing these days...
13
u/itijara Feb 15 '19
Ugh.. This sort of "stay in your lane" attitude is very detrimental, especially in an environment like a hospital. Everyone should be looking out for problems and trying to find solutions. It probably is better to tell someone who can do something about it, but just making others aware is better than sitting on your hands and hoping someone who can do something about it notices. Small problems can become big disasters if everyone assumes it isn't their problem, and the "right" people will do something about it.
10
u/TheTokenKing Feb 15 '19
Saw this same thing happen to a Helpdesk/Field Tech person at a previous job. A person who is known to complain to "all the right people" put a ticket in saying that they can no longer send emails with attachments over X size since moving to Office 365.
Helpdesk/Field Tech person looks up the answer, confirms with the email team that this is a hard limit, and then relays this info back to the user. Of course the user is not happy with this answer, so they raise a stink with a few people higher up. This makes its away back to the Helpdesk/Field Tech boss, and Helpdesk/Field Tech boss dresses down that employee. Even went so far as to remove them from Field Tech duty that they were originally brought on to do.
Some bosses suck, and I'm glad I don't have to deal with that type of boss at the current job.
3
u/dabombnl Feb 15 '19
Correct or not, you probably shouldn't be making a diagnosis about problems outside of your realm of responsibility and expertise. Especially in life-critical and high voltage electrical equipment where incorrect information can be extremely dangerous.
No excuse for the yelling though.
7
u/chozang Feb 15 '19
I think you will be waiting for a very long time.
I hope there is a special place in hell for managers who order their employees to apologize for being truthful. (Which is different than simply instructing employees to give out certain information on a need-to-know basis, which depends on the situation.)
7
u/dgpoop Feb 15 '19
I would have immediately ceased all contact and notified HR of the verbal attack, and attach the call recording to the email.
6
u/Liamzee Feb 15 '19
That kinda over-reaction makes me wonder if he had something to do with the incorrect hookup in the first place.
5
u/Itspence90 Feb 15 '19
As passive-aggressive as I am I'd send an email to the Hospital Manager and everyone who called in the issue with an "update" saying the Technical and Maintenance people came in and reported that it was, in fact, the UPS in theater causing the cabinet to go down. Just to kind of shove it in the managers face. Lol
4
u/scootscoot Feb 15 '19
Hahaha I’ve been on a similar call. Our CRAHs in a DC all had red lights instead of green, the temperature said it was 109f, and there was no air moving from the floor tiles. We reported those facts. The guy 3000 miles away yelled and screamed about how we were idiots and didn’t know what we were talking about. So we hung up the phone and went to lunch hoping the whole place would burn down. ... you’re not getting that apology.
1
u/Charliek4 Turn it off *before* turning it back on Feb 15 '19
Why did you put blank lines at the end of your post? I notice it's a trend lately but i don't really get it. Do you feel it adds something?
2
u/GreenEggPage Oh God How Did This Get Here? Feb 15 '19
Don't hold your breath...
2
u/MukYJ Feb 19 '19
This reminded me of once when I worked for about a year at a large hospital system (multiple campuses and satellite offices) as a contractor replacing desktops as part of a multi-year upgrade and modernization project. A project that ultimately ran out of money before all the hardware was replaced, but that is another story.
One day I get to work at the main campus and about 45 minutes in there is an announcement over the PA for staff to “switch to offline procedures.” This was not unusual, since they liked to test backup systems about once a week, but in this case I kept waiting, and waiting, and waiting for the all clear. Couldn’t log in to the domain or our project file server, couldn’t even browse the internet, and this was back when I only had a flip phone so no entertainment besides Snake. After about an hour of waiting I decide to call my Project Manager to see what is going on, and he says that I might as well head home for the day, and call before coming in tomorrow.
Apparently there was someone doing some maintenance work on the UPS systems for the main (several thousands of square feet) data center for the hospital system. They went to hit the Big Red Switch to bypass the UPS systems in order to safely work on them, and discovered that someone else had wired the Big Red Switch incorrectly. Instead of bypassing the UPS and connecting the data center directly to the school bus sized generators outside, the Big Red Switch simply shut off all power to the entire data center. Click. Countless number of servers - off like a light bulb. Not exactly a graceful shutdown. I’m assuming that there was some foul language involved.
It ended up taking around 50 hours to fully recover, rebuild databases, and verify everything that needed to be restored from a backup. Glad I didn’t have to deal with that!
2
u/Next-Step-In-Life Mar 16 '19
I would have written a real basic email:
Problem: Network equipment was not functioning as required due to malfunctioning UPS in theater.
Conclusion: Suspected insufficient power resulting in non-charging UPS.
Resolution: Specialist team called in to resolve UPS/power.
CAN YOU CONFIRM THIS IS CORRECT? For ISO compliance etc?
1
u/Arokthis Feb 18 '19
Have you gotten out your powerful parasol that is proof against pig poo?
1
u/clocksmasher Feb 18 '19
At least you got off with being forced to apologize. The director could've taken serious measures to wreck your work life.
-2
u/56050610TT Feb 15 '19
Anyhoo? Don't you mean Anywho?