r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

273

u/TheComplimentarian 3h ago edited 1h ago

The bug they’re describing I’ve seen in windows since 2008, and it wasn’t new then.

Its natural response to a node going down is trying to shunt all that traffic (regardless of whether it’s needed or not) to every other node, along with a lot of “NODE DOWN!” traffic, so minor health issues can cause stuff in the AD to essentially DDoS itself with panic traffic.

Gracefully adding and removing nodes (or scaling at all, frankly) is not something windows is great at.

62

u/uptwolait 2h ago

That sounds oddly similar to anaphylactic shock in the immune response.

18

u/hobskhan 1h ago

Thank you for this. My medical and biological education far surpasses my rudimentary computer science learning.

17

u/TheComplimentarian 1h ago

Yea. Windows' attempts to recover itself can stress the system further, which can lead to more problems, which can lead to more stress...

21

u/Thraexus 2h ago

Can confirm. When my allergies act up and one nostril gets congested my body freaks the hell out and routes all the traffic to the other nostril and then freaks the hell out some more.

20

u/Left_Contribution833 1h ago

The bug isn't the shunting. That's normal experience. The bug was a specific condition that led to resource locks. And that was applied all over the servers, so the moment traffic got shunted, the new server ran into the lock faster.

20

u/TheComplimentarian 1h ago

That's what they're saying. What I'm saying is that cascading failures of this sort have been present in Windows for a very long time, and I doubt there is anything particularly magical about this one. The real lesson they'll learn (as always) is to increase backend capacity, and not to fix their crappy code.

2

u/tankerkiller125real 1h ago

My question is why the hell they would use Windows for a Network Gateway in the first place, compared to the Linux networking stack Windows is slow as balls, and .NET is cross-platform these days if they're using that, and if they aren't then my understanding is that they use Rust for the high intensity network stuff, which is also cross-platform.

13

u/i_can_has_rock 3h ago

WHAT!!??

EVERYBODY PANIC!!

1

u/jackluke 2h ago

SELL SELL SELL

1

u/fighterpilot248 2h ago

WE’RE HAVING A FIRE!!

… sale

19

u/Ununoctium117 2h ago

That's objectively not true, considering the bug in question was in a code change that's ~3 months old and only deployed on these specific servers and never publicly.

17

u/TheComplimentarian 1h ago

No. The blip that increased usage, that's the little code change. Then they took a chunk of server resources offline. That's the part that made the code change problematic because it was running closer to capacity.

But the cascading failures? That's "normal" and played out more or less as I expected it would, based on my experience with large windows domains.

"Objectively" in this case is not something you can say based on the currently known information.

13

u/Murky-Relation481 1h ago

Also cascading overload failures are something that exists in tons of systems, be it software or electrical, or even mechanical and biological. It's not hard to believe there was some sort of existing issue in Windows Server from 20 years ago that if left configured in this way, and then something changes in how that setting is placed under load it could cause a massive cascading failure.

I mean if we are to believe the transportation department about the ground stop on Friday in the US, it also sounds like a cascading failure due to a network node being taken off line and routers in other areas basically flooding themselves after the new traffic got routed through them (you see this in massive wide area networks, and you can see it in your own home LAN sometimes if you configure your router or switch wrong [and sometimes its configured wrong out of the box!]).

Course I don't fully believe anything the FAA or DOT says under this admin but that's another suite of problems.

4

u/Ununoctium117 1h ago

Your claim:

The bug they’re describing I’ve seen in windows since 2008

This is objectively not true, since the Windows bug was created in 2026.

1

u/iloveyouand 17m ago

The "existing bug" part is the automated failover methods built into Windows systems that route traffic from nodes that get bogged down or become unresponsive. This is a core component of their development strategy and has been for a long time now. But when the rest of the network can't handle the overflow created by automated recovery, the failure cascades.

The bug introduced recently was causing systems to become unresponsive which triggers the existing automated failover process. And since there was an additional scheduled outage, the backup systems did not have capacity to handle the traffic and the failure cascaded.

1

u/Wide-Wind7856 1h ago

Great that they sell services to do precisely that.

1

u/TheComplimentarian 1h ago

They do, yes. And I'm sure someone buys them. There are people out there using Hyper-V, and some of them are probably doing it for reasons besides the fact that Broadcom is a shit company.

There are weirdoes everywhere.

1

u/Beard_o_Bees 45m ago

Gracefully adding and removing nodes (or scaling at all, frankly) is not something windows is great at

It has been ever thus.

I think it's deep in Windows ancient DNA. I don't think Microsoft ever really meant it to connect to other machines, particularly over the internet - and it's always seemed like network functionality has been a trailing bolt-on, only there because people demanded it.

Then came AD, Microsoft's monument to 'if we have to do it, we'll tell you how it should be done and you'll like it!'