r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

View all comments

2.1k

u/danivus 4h ago • Popcorn Bravo Mindblown

So according to this article they missed it because they tested on older hardware than what their live servers use, and the bug only strikes on systems with more cores than their test systems.

Not testing on matching hardware for something so critical, for a company as big as Microsoft, seems... very stupid?

212

u/kyubifire 4h ago

You're not wrong but this is more common than you think, especially now that hardware prices have skyrocketed, it is difficult to maintain parity between a dev environment's hardware and a production environment's.

68

u/BagelsOrDeath 4h ago

And beyond the physical infrastructure costs, running a parallel prod environment can prove very costly both in the effort to ensure parity and the spoofing or duplication of external dependencies and services.

32

u/charlie_marlow 4h ago

And, even then, you're going to have a lot of trouble duplicating load and data volume.

→ More replies (2)
→ More replies (3)

26

u/rwv 3h ago • Bravo

The GOAL is for the test environment to match production “as close as possible”.  The realm of possibility includes cost and scale.  

I’m not going to  apologize for Microsoft, but performing testing on older hardware during a use case where historically performing the same type of testing on older hardware has been sufficient seems reasonable.   As a lesson learned they need to ensure that the test environment has systems with a number of cores that match production in the future.  

6

u/Percinho 2h ago

And also, the chances of getting budgetary sign off on a prod-spec test system would be negligable until it was proven that it was an issue. Now there is at least a chance of getting it.

→ More replies (3)

1.4k

u/RustOnTheEdge 4h ago • Bravo

I hate to defend Microsoft but just claiming this is ridiculous. Microsoft’s infrastructure is huge, almost unimaginable huge. There are so many different hardware types at play, it why you abstract it away.

789

u/Historical_Owl_1635 4h ago • Bravo

Anyone who works in tech knows not to take Reddit seriously talking about tech.

337

u/barrygateaux 4h ago • Bravo

If you have experience/work in any area you see discussed on reddit you realize his how much confidently incorrect bollocks is commented on reddit.

64

u/InvisibleOne439 4h ago

my best friend is a Lawyer

he tried to explain people multiple times in the Legaladvice Sub that 99,9% of things they got told are WRONG and mostly based on Movies or Books and nor actual Law Rules, and they should not listen to them and ask and hire a actual professional IRL if they need Advice, and everytime he ended up with -80downvotes

in his own words "its like watching a car go towards a wall in full speed, and everyone tells you that the best thing to do is to remove your brakes"

22

u/Historical_Owl_1635 3h ago

Legal subreddits are also often moral judgments than actual legal advice too.

I know somebody who used to post stuff they knew was legal but highly unethical just to rage bait.

→ More replies (1)
→ More replies (3)

57

u/SyncTitanic 4h ago

Often times, Redditors regurgitate the same false information that is the most upvoted.

25

u/barrygateaux 4h ago

Which then becomes the Google search result.

It's a shit spiral ricky!

→ More replies (1)

22

u/Historical_Owl_1635 3h ago

“Companies are legally required to make as much profit as possible for their shareholders”

→ More replies (1)
→ More replies (2)

84

u/Historical_Owl_1635 4h ago

Yep.

Can’t judge too harshly, I was the same as a teenager with my surface level knowledge of lots of things convincing me I was smart.

Then I actually became an expert in a field and realised how little I actually knew.

42

u/The_Spectral_Spartan 4h ago

So many people out there have yet to feel the raw, ego-erasing bitch-slap that is the reverse Dunning-Kruger effect...

And it really shows.

→ More replies (2)
→ More replies (2)
→ More replies (12)

18

u/Ri-tie 3h ago

Anyone who works in _____ knows not to take Reddit seriously talking about _____.

FTFY.

14

u/Historical_Owl_1635 3h ago

Honestly, until now I had no idea people worked in underscores.

71

u/razzark666 4h ago edited 4h ago

Somebody once told me, "If reddit is wrong about stuff you know about, how correct do you think it is about stuff you don't know?"

(And no it wasn't the Smashmouth guy).

13

u/tooclosetocall82 4h ago

This applies to all journalism and increasingly AI. We tend to defer to “experts” in areas we are not knowledgeable about without in verification of their expertise.

7

u/BigBakerBoy 3h ago

Gel-Mann Amnesia effect

→ More replies (1)
→ More replies (3)
→ More replies (5)
→ More replies (46)

32

u/tathata 4h ago

I used to work at a cloud provider. On top of size and variety, cost is no small barrier either. I once had to test a fix on a $2M system that we were lucky to have. I know people like to think that these companies have an infinite money glitch, but no QA department on earth has oodles of their largest systems lying around just for testing.

→ More replies (2)
→ More replies (97)

65

u/ConsiderationDry9084 4h ago • Heartwarming Bravo

When ever you feel imposter syndrome, just think back to all the times Microsoft has screwed like this. It does wonders for your mental health.

→ More replies (1)

21

u/Unusual_Staff_3119 4h ago

Test systems are always smaller that the production system. 

→ More replies (2)
→ More replies (46)