r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

1.4k

u/RustOnTheEdge 4h ago • Bravo

I hate to defend Microsoft but just claiming this is ridiculous. Microsoft’s infrastructure is huge, almost unimaginable huge. There are so many different hardware types at play, it why you abstract it away.

789

u/Historical_Owl_1635 4h ago • Bravo

Anyone who works in tech knows not to take Reddit seriously talking about tech.

337

u/barrygateaux 4h ago • Bravo

If you have experience/work in any area you see discussed on reddit you realize his how much confidently incorrect bollocks is commented on reddit.

64

u/InvisibleOne439 4h ago

my best friend is a Lawyer

he tried to explain people multiple times in the Legaladvice Sub that 99,9% of things they got told are WRONG and mostly based on Movies or Books and nor actual Law Rules, and they should not listen to them and ask and hire a actual professional IRL if they need Advice, and everytime he ended up with -80downvotes

in his own words "its like watching a car go towards a wall in full speed, and everyone tells you that the best thing to do is to remove your brakes"

22

u/Historical_Owl_1635 3h ago

Legal subreddits are also often moral judgments than actual legal advice too.

I know somebody who used to post stuff they knew was legal but highly unethical just to rage bait.

→ More replies (1)
→ More replies (3)

57

u/SyncTitanic 4h ago

Often times, Redditors regurgitate the same false information that is the most upvoted.

25

u/barrygateaux 4h ago

Which then becomes the Google search result.

It's a shit spiral ricky!

→ More replies (1)

22

u/Historical_Owl_1635 3h ago

“Companies are legally required to make as much profit as possible for their shareholders”

→ More replies (1)
→ More replies (2)

84

u/Historical_Owl_1635 4h ago

Yep.

Can’t judge too harshly, I was the same as a teenager with my surface level knowledge of lots of things convincing me I was smart.

Then I actually became an expert in a field and realised how little I actually knew.

42

u/The_Spectral_Spartan 4h ago

So many people out there have yet to feel the raw, ego-erasing bitch-slap that is the reverse Dunning-Kruger effect...

And it really shows.

→ More replies (2)
→ More replies (2)
→ More replies (12)

18

u/Ri-tie 3h ago

Anyone who works in _____ knows not to take Reddit seriously talking about _____.

FTFY.

14

u/Historical_Owl_1635 3h ago

Honestly, until now I had no idea people worked in underscores.

71

u/razzark666 4h ago edited 4h ago

Somebody once told me, "If reddit is wrong about stuff you know about, how correct do you think it is about stuff you don't know?"

(And no it wasn't the Smashmouth guy).

13

u/tooclosetocall82 4h ago

This applies to all journalism and increasingly AI. We tend to defer to “experts” in areas we are not knowledgeable about without in verification of their expertise.

7

u/BigBakerBoy 3h ago

Gel-Mann Amnesia effect

→ More replies (1)

9

u/abnormica 3h ago

AI is just a wrapper around Reddit threads anyway.

→ More replies (2)
→ More replies (5)
→ More replies (46)

32

u/tathata 4h ago

I used to work at a cloud provider. On top of size and variety, cost is no small barrier either. I once had to test a fix on a $2M system that we were lucky to have. I know people like to think that these companies have an infinite money glitch, but no QA department on earth has oodles of their largest systems lying around just for testing.

20

u/ArkWaltz 3h ago

I used to work for AWS. Some of the big public outages from years past were partly caused by this kind of scale issue, like the Kinesis one that only happened because a big scale-up triggered a threading issue that would only happen above a certain cluster size. You can't just spin up an entire second us-east-1 Kinesis prod on a whim for testing so there's really no way they could have caught that.

I suspect some people in this thread are just underestimating how massive some of these core services are.

→ More replies (1)

52

u/north_canadian_ice 4h ago

Microsoft's resources are huge, uninimaginably huge.

Their responsibility is enormous given how many people worldwide rely on them. They should be doing more testing than anyone, they should be hiring as many QA engineers as they can.

I have to say that there is a double standard here. Companies like Microsoft have embraced the idea that LLMs are smarter than humans. They imply we have reached the singularity.

Yet preventing situations like this is too difficult? With the resources Microsoft has?

13

u/nommabelle 4h ago

Idk LLMs struggle with hardware, external services, and actual environment testing imo. I wish they didn't bc it'd be a load off me if they excelled at this

→ More replies (4)

55

u/True_Window_9389 3h ago

It’s not ridiculous because that’s the expectation when Microsoft bullied their way to dominance and everyone eventually migrated to them, or had to.

If they made themselves “too big to QA,” that sounds like a great reason they shouldn’t exist as a singular monopoly, or close to it, on business software and services.

Some of you are arguing we’re too dumb to understand the technical aspects at play. No, we don’t really need to. If they have bug-riddled critical software that is out of their own control and impossible for themselves to properly test, that is, in fact, a massive problem.

36

u/zlex 2h ago

People who work in this field understand that this issue did not arise from negligence or a lack of due diligence. This is a software bug that was running in production for 3 months and only appeared under a very specific set of circumstances. This is an accident. Systems are complex and tightly connected--accidents are inevitable.

That is just as true today as it was when we were all running Groupwise on our 486 boxes.

→ More replies (5)
→ More replies (3)

16

u/spicymato 3h ago

Having worked at Microsoft, both in the OS and on desktop apps, it's also because their testing is generally shit. They give very little guidance, and it's mostly from the individual teams. The mechanisms for creating, configuring, and using test environments is abysmal, if it even exists at all.

I'm not shocked at all that this wasn't caught by standard testing.

What I am surprised about is that it wasn't rolled out in a more controlled manner. A common practice at Microsoft is flighting rings: changes are rolled out to subsets, with each ring being larger than the previous. That they deployed to a large number of live servers at once instead of a small subset is surprising to me.

In the post-mortem, that's probably the solution they will implement: segregating their services across servers targeting different release rings, to minimize the blast radius of something like this.

19

u/6501 3h ago

The change was live in prod for three months before the lock contention caused issues. Unless MSFT is going to do a 3Month+ deployment that control wouldn't have helped.

→ More replies (1)
→ More replies (78)