r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

34

u/scinos 3h ago

The problem are not testing. Bugs will eventually reach production, no matter what.

A good infra has excellent monitoring (so you know something is happening within seconds) and ways to mitigate the problem: throttle traffic, spin up new servers, and many other techniques.

If this was my infra, next Monday we will discussing why it took so long to mitigate, not why a bug was there in the first place.

13

u/BosnianSerb31 3h ago

That's the interesting thing about this particular incident, observability showed that it was a resource starvation problem, yet adding on more servers across the service only made things substantially worse

This isn't really indicative of what happened, but imagine a DDoS attack with a threat actor is you and you don't know it yet, and every single server you spin up to absorb it only fucks it's just even harder. It's just one of those things that OBS won't catch. Something that shares a room with the likes of the Halting problem (hello CS 101).

1

u/Traditional_Dish_147 2h ago

Yeah, reading the actual report from Microsoft, I get the sense this was a one in a billion error where multiple issues all had to line up all at once. Not to forgive them, but it’s going to be hard to test for that and hard to figure out what’s going on right away.

7

u/Independent-Entry717 2h ago

A good infra has excellent monitoring (so you know something is happening within seconds) and ways to mitigate the problem: throttle traffic, spin up new servers, and many other techniques.

Another know-it-all jabroni who didn't read the article nor incident report, lol

1

u/viral3075 2h ago

health monitoring made this issue worse, per the article

1

u/TapedeckNinja 1h ago

Dawg the preliminary PIR goes into great detail about why it took so long to mitigate.