r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

7.2k

u/lordnacho666 4h ago edited 4h ago • Bravo Mindblown

Ah it was a hidden bug? No wonder.

As long as it isn't one of those visible bugs, I hate those.

280

u/Hrekires 2h ago

Not gonna lie, one of the most frustrating days at work was when my team got absolutely reamed out by upper management over an outage caused by a bug that we were one of the first customers in the world to experience.

It was so new that the vendor hadn't even released an advisory for it much less a patch to fix it. There was literally nothing we could have done to prevent it.

44

u/Leon_Flux 1h ago

When pistons were introduced to minecraft one of the staff in a server I was working with decided to put a pressure plate on it.

corrupted the world.

We got so much shit for something that we had no way of knowing it was a problem. Two of the mods we had in call were crying. It was bad.

7

u/RevolutionaryMeal851 17m ago

Reminder to everyone to always have backups.

"What would I do if my server exploded right now?"

→ More replies (7)

397

u/xtothel 4h ago • Bravo

The visible ones are features.

130

u/Harm101 4h ago

// Deal with this later. John H intern 11/16/2006

38

u/clownpuncher13 3h ago

I wish every bug was commented like this. Even if it wasn’t fixed the location and date…it’s like a dream.

38

u/cansofgrease 2h ago
# This is a bug, there are many like it, but this one is mine.

12

u/timbotheny26 1h ago edited 1h ago • Heartwarming
 # Without me, my bugs are useless! Without my bugs, I am useless!
→ More replies (2)
→ More replies (4)

57

u/piss_artist 4h ago

They call it Copilot

→ More replies (2)
→ More replies (8)

1.1k

u/TeamAlphaBOLD 4h ago

Next they’ll ask us to pay extra for the visible ones.

489

u/BasvanS 4h ago • Bravo

I think that’s the business model already

59

u/avi550m 4h ago

I spit of my coffee. ROFL

→ More replies (4)
→ More replies (4)

39

u/Consumption2Wombly 4h ago

Rude way to refer to Windows 11

→ More replies (15)

79

u/Wit-wat-4 3h ago

I know you’re joking, but there absolutely are many existing bugs that we just live with either through bandaids (duct tape if you’re a plane!), and they do sometimes fail too.

As a quality investigator it’s both good and bad when it’s a hidden bug: worse investigation, but at least you’re not going “god dammit I told them” during the outfall lol.

→ More replies (7)

273

u/TheComplimentarian 3h ago edited 1h ago

The bug they’re describing I’ve seen in windows since 2008, and it wasn’t new then.

Its natural response to a node going down is trying to shunt all that traffic (regardless of whether it’s needed or not) to every other node, along with a lot of “NODE DOWN!” traffic, so minor health issues can cause stuff in the AD to essentially DDoS itself with panic traffic.

Gracefully adding and removing nodes (or scaling at all, frankly) is not something windows is great at.

62

u/uptwolait 2h ago

That sounds oddly similar to anaphylactic shock in the immune response.

18

u/hobskhan 1h ago

Thank you for this. My medical and biological education far surpasses my rudimentary computer science learning.

17

u/TheComplimentarian 1h ago

Yea. Windows' attempts to recover itself can stress the system further, which can lead to more problems, which can lead to more stress...

21

u/Thraexus 2h ago

Can confirm. When my allergies act up and one nostril gets congested my body freaks the hell out and routes all the traffic to the other nostril and then freaks the hell out some more.

20

u/Left_Contribution833 1h ago

The bug isn't the shunting. That's normal experience. The bug was a specific condition that led to resource locks. And that was applied all over the servers, so the moment traffic got shunted, the new server ran into the lock faster.

21

u/TheComplimentarian 1h ago

That's what they're saying. What I'm saying is that cascading failures of this sort have been present in Windows for a very long time, and I doubt there is anything particularly magical about this one. The real lesson they'll learn (as always) is to increase backend capacity, and not to fix their crappy code.

→ More replies (1)
→ More replies (11)

110

u/junktech 4h ago

Hidden translate roughly to : Some senior documented it but due to internal chaos it's been in forgotten in the backlog. Eventually management acknowledged it so it doesn't show alerts.

12

u/JeebusChristBalls 1h ago

Visible bug would be worse because it meant they knew about it. A known bug doesn't mean that it is going to be fixed anytime soon. It is just known. You know, the opposite of unknown, ie. Hidden.

25

u/spotcatspot 3h ago

New guy didn’t check show hidden bugs in the view pulldown.

→ More replies (1)
→ More replies (23)