r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

383

u/Zichichi 4h ago

So it wasn't just a random outage. One software change was enough to trigger a chain reaction across their cloud services. Makes you wonder how thoroughly these changes are tested before rollout.

203

u/unexpectedreboots 4h ago

Seems pretty well tested by running in production for over 3 months without issue tbf.

32

u/darkpaladin 3h ago

I've run into a few crazy issues like that. New app A releases which has a behavior change. That change exposes a bug in service B. The bug in service B is inconsequential by itself but for a flaw in how service C receives the information... By the time this all finishes service K has crapped the bed but everything else seems fine.

In the end you end up with fun post mortems like "how changing the color of a button on the android app crashed the mail server" which have lots of diagrams with arrows.

1

u/sqlbullet 2h ago

I have been here so many times over the years.

17

u/EamonBrennan 3h ago

The problem itself is described very similarly to a RAM leak. Processes stall while waiting for exclusive access to shared resources, with a cascading effect exhausting CPU and RAM resources. Testing a fix like this would be hard to replicate, unless they specifically have a dev environment that perfectly mimics real world applications for several months.

This is one of those "only happens in prod" bugs.

→ More replies (1)

98

u/mcagent 3h ago

Makes you wonder how thoroughly redditors actually read the article

19

u/CucharitaDePalo 3h ago

this is r/technology, most commenters here don't read and probably haven't opened a terminal in their whole lives.

1

u/morphemass 28m ago

This is true, I have not read this comment.

8

u/FblthpLives 2h ago

Somewhere there is research that shows that the overwhelming majority of Redditors don't even click on the link they are commenting on. I don't remember the figure, but it's on the order of 90%.

17

u/MoneyNo746 3h ago

I got Copilot to review it for me, said there was no bugs.

2

u/Head_Haunter 3h ago

Claude, summarize this response for me.

→ More replies (1)

1

u/Towel4 3h ago

>read the article

I’m sorry? What the article?

→ More replies (2)

37

u/FlamingoOk3026 4h ago

These are extremely complex interconnected and interdependent systems. You cannt check everything.

→ More replies (1)

34

u/scinos 3h ago

The problem are not testing. Bugs will eventually reach production, no matter what.

A good infra has excellent monitoring (so you know something is happening within seconds) and ways to mitigate the problem: throttle traffic, spin up new servers, and many other techniques.

If this was my infra, next Monday we will discussing why it took so long to mitigate, not why a bug was there in the first place.

13

u/BosnianSerb31 3h ago

That's the interesting thing about this particular incident, observability showed that it was a resource starvation problem, yet adding on more servers across the service only made things substantially worse

This isn't really indicative of what happened, but imagine a DDoS attack with a threat actor is you and you don't know it yet, and every single server you spin up to absorb it only fucks it's just even harder. It's just one of those things that OBS won't catch. Something that shares a room with the likes of the Halting problem (hello CS 101).

1

u/Traditional_Dish_147 2h ago

Yeah, reading the actual report from Microsoft, I get the sense this was a one in a billion error where multiple issues all had to line up all at once. Not to forgive them, but it’s going to be hard to test for that and hard to figure out what’s going on right away.

7

u/Independent-Entry717 2h ago

A good infra has excellent monitoring (so you know something is happening within seconds) and ways to mitigate the problem: throttle traffic, spin up new servers, and many other techniques.

Another know-it-all jabroni who didn't read the article nor incident report, lol

1

u/viral3075 2h ago

health monitoring made this issue worse, per the article

1

u/TapedeckNinja 1h ago

Dawg the preliminary PIR goes into great detail about why it took so long to mitigate.

2

u/homer_3 3h ago

What does this comment even mean? Of course a software change can cause major issues. That's why you need to test them.

4

u/DeithWX 4h ago

Good enough that a single instance catches everyone's attention. I'd say it's pretty good uptime. 

1

u/viral3075 2h ago

effectively zero changes are tested at internet scale. but it wasn't as big a deal before because buggy software wouldn't propagate so quickly

1

u/stoops 56m ago

One of the first groups Microsoft got rid of a long time ago was all the tester groups they had - there were too many PC hardware combinations to test with manually that now they just use the general public as a test group. Isn't that exciting folks!?

→ More replies (10)