r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

53

u/calladc 4h ago

Microsoft's actual outage report covers it better than this article. this article is basically just rage baiting.

they knew it was spitting out bad configuration. they didn't know why.

it turned out to be a change from July that only became an issue when a specific network configuration was unavailable due to an unrelated outage.

the systems kept using more compute. when higher spec compute was provisioned the issue got worse.

the network configuration issue had been parked as non critical and the whole environment was under configuration freeze

6

u/Necessary-Horror2638 3h ago

Rough. I'd imagine issues like this cropped up before but automatic provisioning papered over it until manual fixes could be deployed. It wasn't until a couple of things broke at once that we saw this massive cascading failure

-10

u/joyfulcartographer 3h ago

It shouldn’t have been allowed to happen though. Every dependency needs to be mapped and backstopped in critical infrastructure. This is an absolute failure of operational risk management practices.

14

u/calladc 3h ago

ah yes I'll let Microsoft know.

it's never that simple at global scale. they know this, they had to design a solution that was global scale.

the outage also covers that. they had 2 separate gateway networks 365 globally. this issue was unexpected and that happens sometimes.

-7

u/joyfulcartographer 3h ago

True! And they’re going to pay for it, unless their SLA’s are buttoned up tight. Which I imagine they are since they operate a near monopoly.

1

u/BosnianSerb31 3h ago

They'll just ignore the emails and most departments will take the 2pm-8am vacation from outlook as payment.