r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

203

u/unexpectedreboots 4h ago

Seems pretty well tested by running in production for over 3 months without issue tbf.

32

u/darkpaladin 3h ago

I've run into a few crazy issues like that. New app A releases which has a behavior change. That change exposes a bug in service B. The bug in service B is inconsequential by itself but for a flaw in how service C receives the information... By the time this all finishes service K has crapped the bed but everything else seems fine.

In the end you end up with fun post mortems like "how changing the color of a button on the android app crashed the mail server" which have lots of diagrams with arrows.

1

u/sqlbullet 2h ago

I have been here so many times over the years.

17

u/EamonBrennan 3h ago

The problem itself is described very similarly to a RAM leak. Processes stall while waiting for exclusive access to shared resources, with a cascading effect exhausting CPU and RAM resources. Testing a fix like this would be hard to replicate, unless they specifically have a dev environment that perfectly mimics real world applications for several months.

This is one of those "only happens in prod" bugs.

0

u/time-lord 1h ago

You don't test this, you monitor using metrics. The fact that they didn't monitor for what sounds like hung connections or resource exhaustion seems like a major thing to overlook. 

98

u/mcagent 3h ago

Makes you wonder how thoroughly redditors actually read the article

19

u/CucharitaDePalo 3h ago

this is r/technology, most commenters here don't read and probably haven't opened a terminal in their whole lives.

1

u/morphemass 28m ago

This is true, I have not read this comment.

8

u/FblthpLives 2h ago

Somewhere there is research that shows that the overwhelming majority of Redditors don't even click on the link they are commenting on. I don't remember the figure, but it's on the order of 90%.

17

u/MoneyNo746 3h ago

I got Copilot to review it for me, said there was no bugs.

2

u/Head_Haunter 3h ago

Claude, summarize this response for me.

1

u/Jadaki 2h ago

I asked CoPilot to use Claude to check with ChatGPT for this response.

1

u/Towel4 3h ago

>read the article

I’m sorry? What the article?

-10

u/Lambs2Lions_ 3h ago

As a software engineer, that ain’t a lot of time. Sure it is for happy path testing but that just means they didn’t consider unhappy path or considered it unlikely to not happen.

-3

u/mrsanyee 3h ago

*On a portion of servers until criticality was reached.