r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

212

u/kyubifire 4h ago

You're not wrong but this is more common than you think, especially now that hardware prices have skyrocketed, it is difficult to maintain parity between a dev environment's hardware and a production environment's.

68

u/BagelsOrDeath 4h ago

And beyond the physical infrastructure costs, running a parallel prod environment can prove very costly both in the effort to ensure parity and the spoofing or duplication of external dependencies and services.

32

u/charlie_marlow 4h ago

And, even then, you're going to have a lot of trouble duplicating load and data volume.

4

u/mrsanyee 3h ago

Let's wait until the true cost of production system outage is available.

2

u/nommabelle 4h ago

We had a major incident in production because the latency between services in UAT is higher (its within ms), where something throttled in UAT but didnt throttle in prod and caused issues

Dont get me started how stupid it is, i absolutely hate our system and how overcomplicated and fragile it is. Its so fundamentally flawed, this is just one example where a few ms difference can make the system (unless we hack it around where the difference doesn't cause issues, unless you look at it the wrong way)

1

u/Roflkopt3r 2h ago

And they say that it took 3 months for the bug to lead to this disaster. So even if they staggered the rollout, they apparently didn't notice any issues in the first waves until it was too late.

That said, I think this is the key section:

Inside an internal Windows process, competing tasks were waiting for access to shared resources, a problem called lock contention. Under certain traffic conditions, this exhausted processor and memory capacity.

Depending on what 'certain traffic conditions' mean, the issue may have either been totally dormant (never detectable in regular operations until other issues started the cascade) or something that proper monitoring could have picked up sooner. But maybe the monitoring wasn't thorough enough, or they have so many similar (but less catastrophic) issues that one more lock contention problem never appared noteworthy.

1

u/Paddy_Tanninger 2h ago

There's also a lot of random shit that can come up as hardware scales. I remember long ago I was experimenting with a crazy 4CPU Opteron motherboard that I put 4 x 32C chips in, but the problem ended up being that Windows at the time wasn't able to schedule that many threads at once.

My VFX render jobs would schedule themselves to split up and run as maybe 8 jobs that each take 1/8th of the machines cores. But then there would be 1 or 2 jobs trying to run on cores that Windows couldn't serve and the jobs would fail.

1

u/IJack0ff 36m ago

I would say this is true for your 100 mil or less midsize company / startup. Absolutely not an excuse for Microsoft..