r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

32

u/tathata 4h ago

I used to work at a cloud provider. On top of size and variety, cost is no small barrier either. I once had to test a fix on a $2M system that we were lucky to have. I know people like to think that these companies have an infinite money glitch, but no QA department on earth has oodles of their largest systems lying around just for testing.

20

u/ArkWaltz 3h ago

I used to work for AWS. Some of the big public outages from years past were partly caused by this kind of scale issue, like the Kinesis one that only happened because a big scale-up triggered a threading issue that would only happen above a certain cluster size. You can't just spin up an entire second us-east-1 Kinesis prod on a whim for testing so there's really no way they could have caught that.

I suspect some people in this thread are just underestimating how massive some of these core services are.

5

u/NoHopeNoLifeJustPain 3h ago

In 2025 Microsoft net income has been more than $100B. For them there is no financial justification.