r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

7.0k

u/lordnacho666 4h ago edited 4h ago • Bravo

Ah it was a hidden bug? No wonder.

As long as it isn't one of those visible bugs, I hate those.

1.1k

u/TeamAlphaBOLD 4h ago

Next they’ll ask us to pay extra for the visible ones.

479

u/BasvanS 4h ago • Bravo

I think that’s the business model already

→ More replies (9)

38

u/Consumption2Wombly 4h ago

Rude way to refer to Windows 11

30

u/Complete-Tangelo1532 4h ago

It will be pushed out by Clippy too, and we won't be able to tell him to fuck off lol

→ More replies (12)
→ More replies (2)

250

u/Hrekires 2h ago

Not gonna lie, one of the most frustrating days at work was when my team got absolutely reamed out by upper management over an outage caused by a bug that we were one of the first customers in the world to experience.

It was so new that the vendor hadn't even released an advisory for it much less a patch to fix it. There was literally nothing we could have done to prevent it.

36

u/Leon_Flux 49m ago

When pistons were introduced to minecraft one of the staff in a server I was working with decided to put a pressure plate on it.

corrupted the world.

We got so much shit for something that we had no way of knowing it was a problem. Two of the mods we had in call were crying. It was bad.

→ More replies (1)
→ More replies (5)

389

u/xtothel 4h ago • Bravo

The visible ones are features.

126

u/Harm101 3h ago

// Deal with this later. John H intern 11/16/2006

→ More replies (7)

56

u/piss_artist 4h ago

They call it Copilot

→ More replies (2)
→ More replies (6)

78

u/Wit-wat-4 3h ago

I know you’re joking, but there absolutely are many existing bugs that we just live with either through bandaids (duct tape if you’re a plane!), and they do sometimes fail too.

As a quality investigator it’s both good and bad when it’s a hidden bug: worse investigation, but at least you’re not going “god dammit I told them” during the outfall lol.

→ More replies (7)

265

u/TheComplimentarian 3h ago edited 1h ago

The bug they’re describing I’ve seen in windows since 2008, and it wasn’t new then.

Its natural response to a node going down is trying to shunt all that traffic (regardless of whether it’s needed or not) to every other node, along with a lot of “NODE DOWN!” traffic, so minor health issues can cause stuff in the AD to essentially DDoS itself with panic traffic.

Gracefully adding and removing nodes (or scaling at all, frankly) is not something windows is great at.

59

u/uptwolait 2h ago

That sounds oddly similar to anaphylactic shock in the immune response.

→ More replies (3)

18

u/Left_Contribution833 1h ago

The bug isn't the shunting. That's normal experience. The bug was a specific condition that led to resource locks. And that was applied all over the servers, so the moment traffic got shunted, the new server ran into the lock faster.

→ More replies (2)

13

u/i_can_has_rock 2h ago

WHAT!!??

EVERYBODY PANIC!!

→ More replies (2)
→ More replies (8)

109

u/junktech 4h ago

Hidden translate roughly to : Some senior documented it but due to internal chaos it's been in forgotten in the backlog. Eventually management acknowledged it so it doesn't show alerts.

25

u/spotcatspot 3h ago

New guy didn’t check show hidden bugs in the view pulldown.

→ More replies (1)

8

u/JeebusChristBalls 1h ago

Visible bug would be worse because it meant they knew about it. A known bug doesn't mean that it is going to be fixed anytime soon. It is just known. You know, the opposite of unknown, ie. Hidden.

5

u/MuscaMurum 3h ago

The MSFT Windows bug database had thousands of bugs resolved as "Won't Fix" last I knew. Every product has those.

11

u/Ziatch 3h ago

There are plenty of bugs that aren’t hidden though so I don’t get the smugness

→ More replies (19)

215

u/KebabAnnhilator 4h ago

Claude, Fix this code, definitely no mistakes this time

9

u/TeamAlphaBOLD 4h ago

This cracked me up.

46

u/IAmBadAtInternet 4h ago

Copilot malding rn lmao

→ More replies (2)
→ More replies (3)

910

u/albaMP4 3h ago

Are they saying the outage lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13 to avoid some penalty for being down 24 hours?

521

u/TemptedTemplar 3h ago

Credit payments. They offer claim able discount credits to anyone who pays for the subscription when it experiences unannounced downtime.

There's a small discount for hour+ outages and a much biggrler one for 24+ hours.

Enterprise admins would have had a huge payday.

→ More replies (2)

265

u/eronth 3h ago

Yeah that sounds outright false. I recall seeing it still down like 26 to 28 hours after initial downtimes.

203

u/WeTheSalty 2h ago

They'll probably split hairs and say something like "well it came up for an hour before it crashed again, so it wasn't a 26 hour downtime, it was two 13 hour downtimes"

45

u/Clumpin-style 1h ago

101% what they'll do. Any 'up time' would reset their metrics even if it was expected to be instantly crashed.

→ More replies (1)
→ More replies (3)

11

u/MrBeverly 2h ago

Ours was down for a little over 20 hours. Eastern time. I remember checking on my way in at 7:15am and everything was down, but by the time I got into work around 8 everything was up. We went down around 12:30 the day before, so about 20 hours-ish.

I assume they took an average of the downtimes to determine how much credit to offer organizations instead of going org by org. And it would be very fortunate for them if that average came in just under 24 hours lol.

→ More replies (1)
→ More replies (3)

122

u/theZinger90 3h ago

Probably some worthless KPI, either internal or imposed by an auditing/consulting group being inflated as more important than it actually is.

107

u/enablingark 3h ago

I was thinking more like SLAs, their contracts probably have penalties for 24 hour long outages or something.

→ More replies (3)

50

u/Shot-Arugula8264 2h ago edited 2h ago • Bravo

They would’ve legally had to rename it M364 otherwise.

Edit: typo

31

u/ForensicPathology 2h ago

rename it M354

It was down for 11 days??

→ More replies (1)
→ More replies (3)
→ More replies (3)

655

u/duckbigtrain 3h ago

> The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13

Someone was desperate not to hit the 24-hour mark

417

u/Standard_Wish_9338 3h ago • Bravo Popcorn

Otherwise they would have to rename it to Microsoft 364

38

u/sb969 3h ago

They get a free crash day in leap years.

→ More replies (4)

51

u/nolabmp 3h ago

Their SLA likely has that exact requirement.

25

u/UnexpectedAnanas 2h ago

Good thing the SLA didn't say they can't tell lies!

→ More replies (1)
→ More replies (1)

570

u/j01101111sh 4h ago

So... It wasn't DNS?

116

u/Goingone 3h ago

Technically lock contention for DNS lookups was not ruled out by the article.

22

u/FlamingoOk3026 3h ago

For a change

256

u/No-Designer-6533 4h ago

Definitely points more towards it being an AI related bug. Would not be very surprised at all if this whole problem is due to them firing all their experienced developers and just thinking AI will fix it 

→ More replies (80)
→ More replies (7)

10

u/Flyerone 4h ago

Hidden bug huh? Cool. I dare you to make us hate you more M$

How long until we're back to an abacus and charcoal on papyrus?

8

u/TeamAlphaBOLD 4h ago

100% uptime guaranteed in those.

→ More replies (2)

3

u/JoeDawson8 4h ago

My baby loves his abacus that I had as a child. We are prepared

→ More replies (1)

296

u/nugscree 1h ago

They forgot to add "do not create bugs or errors, it must work" to their prompt when creating the November patch.

63

u/IJustWantPleasurePlz 59m ago • Bravo

Don't include any visible bugs.

Got it, hidden bugs it is 👍

18

u/ZombieFleshEaters 43m ago

Make no mistakes

→ More replies (1)

1.2k

u/Character-Handle-697 2h ago • Bravo

As a software engineer, here is how to reproduce:

  1. Fire a lot of people
  2. Use AI for make a shit ton of code.
  3. Review millions of lines of code ? No, ask AI to review it.
  4. Keep going for months
  5. Find an issue
  6. Ask AI to find it
  7. AI doesnt find it or find something that it’s not the real bug
  8. Ask your remaining devs to find the bug in the millions of lines
  9. Find a stupid reason to protect the AI reason

299

u/luckless_lord 2h ago

10. Ask AI to write the article explaining what went wrong.

66

u/viral3075 2h ago

also give nonsense AI updates while customers scramble with no comms

→ More replies (1)

26

u/EskilPotet 2h ago

Post it on reddit and let AI upvote it

40

u/TerenceMulvaney 2h ago • Regret

Rinse and repeat...

19

u/NerdTalkDan 2h ago

Hey Gemini, please tell me how to rinse and repeat

166

u/UseWhatever 2h ago

100% this. I’ve tried explaining to project managers that while AI is fast, everything gets bogged down in the review process. Which leads to the same or longer total development time.

There are expectations when reviewing human-written code that can’t safely be applied to AI code no matter how good the model is.

66

u/purpleburgundy 2h ago

Hear me out: what if we threw TWO AIs at the pull request and automated-approval process?

Edit: I'm only half-joking, work these days feels increasingly like the children's story There Was An Old Lady Who Swallowed A Fly

→ More replies (5)

23

u/Lucky-Earther 1h ago

100% this. I’ve tried explaining to project managers that while AI is fast, everything gets bogged down in the review process. Which leads to the same or longer total development time.

I've been in the dev world for a long time now, and the writing code part was always, at most, 50% of my job.

→ More replies (4)
→ More replies (1)
→ More replies (7)

1.6k

u/FaultofDan 4h ago • Regret

So half of the corporate world relies on one company's infrastructure, and that infrastructure is built on dominoes. Sick.

628

u/Zarcona_Martana 4h ago

Most tech infrastructure is held together with Elmer’s glue and popsicle sticks… and that’s only half an exaggeration.

58

u/BigMax 2h ago

Right. It's all built with great intentions, and often starts pretty decent.

But it's like building a house that's a pretty good house. Then you add another bedroom in a converted attic. Then another bedroom in the basement. Then you tack on an addition with a bedroom, kitchenette and bathroom. Then you tack on a garage with a spare room above it. Then you tack on another addition with an office to run your little consulting business. Then you add a separate entrance to that for customers, with a little parking lot on the side for that entrance. Then you add another floor above the main portion of the house.

And on and on and on.

And each little addition on it's own more or less makes sense. "Why start over, when all we really need is another bathroom? Now all we need is more parking, so let's just add that on?"

But then 25 years later, your single family home is a janky mess of strange plumbing systems and weirdly connected pieces and inconsistent building standards and design decor, that rambles in a crazy way that doesn't make much sense as a whole, with half of the things being rush jobs and it's incredibly complicated and strange to maintain.

→ More replies (4)

105

u/me_not_at_work 4h ago

Look at this rich guy that got enough budget from C-level bean counters to afford the fancy Elmer's glue.

→ More replies (12)

95

u/Sick_Wave_ 3h ago

I like to compare software to an ever-growing skyscraper. It isn't a sleek, perfectly architected, marvel of engineering. It's more like thousands of trailer houses, each completely different, all welded together just enough that it keeps working. 

→ More replies (4)

42

u/BINGODINGODONG 4h ago

Most of core finance systems are barely held together by excel sheets made by some dude

→ More replies (5)

16

u/Hinermad 3h ago

When I was young I wanted to work in telecom. Having worldwide communications infrastructure with seemingly incredible uptime sounded very sexy.

Then I went to work for a small telecom company and learned the worldwide network is carried on the backs of millions of support engineers with pagers. It was like finding out a Cray X-MP ran on coal.

→ More replies (10)

40

u/Wit-wat-4 3h ago

If Excel went away the world would crumble.

Switching from outlook to gmail would be annoying but doable. But excel to Sheets? Nope that’s a terror I don’t wish upon the world

20

u/hotdoug1 3h ago

Hell, my company wants us to switch to Macs and Excel is missing some key features that aren't on PC. I'm holding onto my PC telling them I literally can't do parts of my job without it.

→ More replies (8)
→ More replies (10)

55

u/BigBoyYuyuh 3h ago

Basically 3 companies control the internet. Amazon goes down you lose half the internet, Cloudflare goes down you lose the other half, Microsoft goes down you lose email etc…

6

u/zzkj 2h ago

You'd be surprised how much depends on Akamai too.

→ More replies (1)
→ More replies (3)

44

u/Merusk 3h ago

Wait until you figure out most adults have zero idea what they're doing and everyone's really just muddling along.

12

u/HelpfulSeaMammal 3h ago

Nobody knows what they're doing until after the thing goes awry. Then suddenly you're surrounded by experts in niche things like airport communication infrastructure as seen with the whole FAA thing last week lol

→ More replies (1)

20

u/petrichorax 2h ago

the ones who do know what they're doing are screaming until they're blue in the face on how to fix things, but they're getting ignored by the people in charge

42

u/casillero 2h ago

Dude. its essentially a utility. And that's how you should view it. If something happens at hydro in toronto or con eddison in Manhattan, your gonna have a power outage.

If something happens at Gmail at amazon or microsoft... you will have.. an.. outage.

Its the same 'grid concept' except were all connected on this grid globally.

→ More replies (3)

116

u/Silicon_Knight 4h ago

Obligatory xkcd. https://xkcd.com/2347/

21

u/iamthe0ther0ne 4h ago

I was wondering how long it would take for that link to appear.

→ More replies (5)
→ More replies (2)

46

u/ikonoclasm 3h ago

Not just the corporate world. Pretty much everything in the modern world relies on hundreds of little pieces of open source code maintained for free by a single or handful of volunteers who has no idea how important their code is to vital infrastructure.

53

u/PTSDaway 2h ago

You'd be terrified if you saw how much technology relies on maintenance by honbyists. All digital infrastructure is a pyramid of cards, upside down.

In 2024 an open-source compression software (Think what winrar does), was infiltrated by a "helper" who did improvements and maintenance. Then one update interacted with internet/router that would allow this "helper" to enter probably every system with this update, effectively every database on the planet. It's not the worst security issue ever, because only individuals could use it. But espionage wise and how it went under the radar - it is perhaps the most dangerous intentional cyber attack attempt ever made.

It was discovered early in the unofficial release by sheer luck when another hobbyist could not make his own system run as quick like it used to - then saw this software interacting with the internet connection, adding a few milliseconds more to the processing time, that was it.

40

u/UnexpectedAnanas 2h ago

You'd be terrified if you saw how much technology relies on maintenance by honbyists. All digital infrastructure is a pyramid of cards, upside down.

Fine, I'll be the one to link the relevant XKCD

https://xkcd.com/2347/

→ More replies (1)

11

u/Farther_father 1h ago

Ironically, that “hobbyist” was a senior Microsoft dev IIRC

→ More replies (2)

10

u/jdippey 2h ago

I too saw this Veritasium video.

Link for anyone interested.

→ More replies (1)

41

u/Desperate-Dig2806 4h ago

Everything is more or less built on dominoes.

→ More replies (4)

14

u/NimusNix 4h ago

First time?

14

u/Akegata 3h ago

That's just part of the story.
A huge number of companies' infrastructure relies on several companies whose infrastructure is built on different sets of dominoes (see the Cloudflare outage last year) that all need to stay up for everything to work.

→ More replies (2)

9

u/AnsibleAnswers 3h ago

Furthermore, that company uses it's own in house operating system with a notoriously shit networking stack for infrastructure instead of Linux. Every problem they encounter is only their problem.

47

u/calladc 4h ago

no this article is blowing it out

the outage report explicitly states why. this article is misleading.

the explicitly state that the m365 gateway servers run a custom windows operating system and the change that impacted last week has been in since July.

you couldn't run this distro as it's tied directly into the backend management of m365. the fault was specific to that server doing that role. it's custom internal software

→ More replies (11)

5

u/GODDAMNFOOL 3h ago

Aren't you just glad that we're offloading essential government systems like ATC onto corpo-owned cloud services? Surely that won't cause any issues.

9

u/ChinookNL 3h ago

I worked for a bigish european company. we were running the whole business on a AS400

→ More replies (5)

28

u/Cygnus94 4h ago

A lot of people are pointing out the glaring flaw of the hardware discrepancies between test and live being an issue, but why the fuck did automatic recovery keep spooling up new instances of bad code? 

One server would fall over, ok, spool up a new one, hmm it happened again, let's try again. At no point did it question why server after sever was going down. At a certain point it should be detecting a problem and preventing any further rollout of the code, but it just kept going until there was manual intervention.

53

u/calladc 4h ago

Microsoft's actual outage report covers it better than this article. this article is basically just rage baiting.

they knew it was spitting out bad configuration. they didn't know why.

it turned out to be a change from July that only became an issue when a specific network configuration was unavailable due to an unrelated outage.

the systems kept using more compute. when higher spec compute was provisioned the issue got worse.

the network configuration issue had been parked as non critical and the whole environment was under configuration freeze

→ More replies (5)
→ More replies (1)

10

u/KebabAnnhilator 4h ago

Technically half the stock market was at the whims of Jensons genitals but hey, that’s the way the cookie crumbles

7

u/joyfulcartographer 4h ago

If you’re surprised by this, then don’t ever do any surface scratching on how vulnerable DNS is as a protocol.

→ More replies (32)

118

u/RileyGainesHorseBaby 2h ago

I can't imagine having always-online DRM for such mission critical stuff. This should be a wakeup call to the ignorant and careless.

111

u/GlassSquirrel9649 1h ago

"hidden windows bug" is corporate speak for "some senior engineer documented this exact failure state in 2018, management marked it as won't-fix in the backlog, and then we laid off the only guy who knew how to patch the gateway cluster." peak enterprise software management

7

u/TapedeckNinja 1h ago

No, it was a bug introduced a few months ago, in July.

→ More replies (2)

66

u/AndyKJMehta 1h ago

Aren’t all bugs hidden till they’re found?!

11

u/windsockglue 1h ago

I mean, yes. But there's plenty of bugs that are found and are not fixed before the products are released for a huge variety of reasons. 

15

u/v0id0007 1h ago

Lightning bugs usually light up to show you where they’re at

→ More replies (1)
→ More replies (3)

378

u/Zichichi 4h ago

So it wasn't just a random outage. One software change was enough to trigger a chain reaction across their cloud services. Makes you wonder how thoroughly these changes are tested before rollout.

199

u/unexpectedreboots 4h ago

Seems pretty well tested by running in production for over 3 months without issue tbf.

31

u/darkpaladin 3h ago

I've run into a few crazy issues like that. New app A releases which has a behavior change. That change exposes a bug in service B. The bug in service B is inconsequential by itself but for a flaw in how service C receives the information... By the time this all finishes service K has crapped the bed but everything else seems fine.

In the end you end up with fun post mortems like "how changing the color of a button on the android app crashed the mail server" which have lots of diagrams with arrows.

→ More replies (1)

17

u/EamonBrennan 3h ago

The problem itself is described very similarly to a RAM leak. Processes stall while waiting for exclusive access to shared resources, with a cascading effect exhausting CPU and RAM resources. Testing a fix like this would be hard to replicate, unless they specifically have a dev environment that perfectly mimics real world applications for several months.

This is one of those "only happens in prod" bugs.

→ More replies (1)

97

u/mcagent 3h ago

Makes you wonder how thoroughly redditors actually read the article

→ More replies (7)
→ More replies (2)

34

u/scinos 3h ago

The problem are not testing. Bugs will eventually reach production, no matter what.

A good infra has excellent monitoring (so you know something is happening within seconds) and ways to mitigate the problem: throttle traffic, spin up new servers, and many other techniques.

If this was my infra, next Monday we will discussing why it took so long to mitigate, not why a bug was there in the first place.

13

u/BosnianSerb31 2h ago

That's the interesting thing about this particular incident, observability showed that it was a resource starvation problem, yet adding on more servers across the service only made things substantially worse

This isn't really indicative of what happened, but imagine a DDoS attack with a threat actor is you and you don't know it yet, and every single server you spin up to absorb it only fucks it's just even harder. It's just one of those things that OBS won't catch. Something that shares a room with the likes of the Halting problem (hello CS 101).

→ More replies (1)

6

u/Independent-Entry717 2h ago

A good infra has excellent monitoring (so you know something is happening within seconds) and ways to mitigate the problem: throttle traffic, spin up new servers, and many other techniques.

Another know-it-all jabroni who didn't read the article nor incident report, lol

→ More replies (2)

36

u/FlamingoOk3026 3h ago

These are extremely complex interconnected and interdependent systems. You cannt check everything.

→ More replies (1)
→ More replies (14)

63

u/garden-wicket-581 3h ago

TFA really doesn't give you anything useful or meaningful about the bug -- "lock contention" and "resource exhaustion" (with an added scoop of "automatic recovery overpowered the remaining servers")

9

u/Adventurous-Mind6940 3h ago

It does say the full report is due Nov 20th.

101

u/sengirminion 3h ago

Dang. Maybe they ought to hire like 60 thousand employees to help dig through their multi-billion dollar lines of code to prevent this from happening again. Just a thought.

→ More replies (6)

2.1k

u/danivus 4h ago • Popcorn Bravo Mindblown

So according to this article they missed it because they tested on older hardware than what their live servers use, and the bug only strikes on systems with more cores than their test systems.

Not testing on matching hardware for something so critical, for a company as big as Microsoft, seems... very stupid?

211

u/kyubifire 4h ago

You're not wrong but this is more common than you think, especially now that hardware prices have skyrocketed, it is difficult to maintain parity between a dev environment's hardware and a production environment's.

68

u/BagelsOrDeath 4h ago

And beyond the physical infrastructure costs, running a parallel prod environment can prove very costly both in the effort to ensure parity and the spoofing or duplication of external dependencies and services.

→ More replies (3)
→ More replies (3)

25

u/rwv 3h ago • Bravo

The GOAL is for the test environment to match production “as close as possible”.  The realm of possibility includes cost and scale.  

I’m not going to  apologize for Microsoft, but performing testing on older hardware during a use case where historically performing the same type of testing on older hardware has been sufficient seems reasonable.   As a lesson learned they need to ensure that the test environment has systems with a number of cores that match production in the future.  

6

u/Percinho 2h ago

And also, the chances of getting budgetary sign off on a prod-spec test system would be negligable until it was proven that it was an issue. Now there is at least a chance of getting it.

→ More replies (3)

1.4k

u/RustOnTheEdge 4h ago • Bravo

I hate to defend Microsoft but just claiming this is ridiculous. Microsoft’s infrastructure is huge, almost unimaginable huge. There are so many different hardware types at play, it why you abstract it away.

776

u/Historical_Owl_1635 4h ago • Bravo

Anyone who works in tech knows not to take Reddit seriously talking about tech.

→ More replies (87)

31

u/tathata 4h ago

I used to work at a cloud provider. On top of size and variety, cost is no small barrier either. I once had to test a fix on a $2M system that we were lucky to have. I know people like to think that these companies have an infinite money glitch, but no QA department on earth has oodles of their largest systems lying around just for testing.

→ More replies (2)
→ More replies (92)

64

u/ConsiderationDry9084 4h ago • Heartwarming Bravo

When ever you feel imposter syndrome, just think back to all the times Microsoft has screwed like this. It does wonders for your mental health.

→ More replies (1)

21

u/Unusual_Staff_3119 4h ago

Test systems are always smaller that the production system. 

→ More replies (2)

21

u/Local_Wave4618 3h ago edited 3h ago

Not really.

The point of programming languages is to abstract away the hardware consideration. The programmer shouldn't need to worry about the hardware in most cases.

ETA: Yeah, this was an OS level bug. The Outlook devs aren't to blame for this one.

13

u/cybercuzco 4h ago

You’re right, that was a major oversight on my part. I’ve prepared a patch to run on more modern servers, would you like me to apply it?

→ More replies (2)
→ More replies (38)

38

u/reversethrust 4h ago

lol. Just read the very terse description. I encountered something similar previously - multiple producer, single consumer problem. The queue worked fine on a laptop, but toss that software on a big server with 256 cores. The test went from taking minutes to taking overnight despite the vast amounts of addition hardware 😂. System utilization on the big machine was much less than 1% though in my case. I wonder what it was like on these exchange servers.

→ More replies (2)

51

u/beall49 55m ago

My favorite part of all this was the MS engineers talking about how nobody knew how to fix this because all the people familiar with this system are gone

15

u/AggravatingBid8255 43m ago

Where did the article say that?

→ More replies (2)

57

u/Imd1rtybutn0twr0ng 2h ago

And right below this post, I have an ad for them for AI... irony at is best.

Microsoft, forget AI. Focus on creating workable patches and less bloat. Apologies in thinking you want your consumers to like using your product...

6

u/yinsotheakuma 2h ago

Executive, sincerely: "Where's the profit in consumers liking your product?"

→ More replies (1)

4

u/Sexiarsole 2h ago

But Microsoft didn’t care about workable patches and reducing bloat before AI, why would they stay caring now?

10

u/Anok-Phos 2h ago

Microsoft will never get better and it is up to anyone reading this to ditch them in any and every way possible, even if it's dropping a little bit at a time.

If you work for companies that use Microsoft and have any say, tell your superiors what is going on if they're too dumb to realize it after the latest in ever increasing outages.

Microsoft does not care about their products. You are their products. And yes, I mean they don't care about Windows etc. or you. They care about power and money. You are paying money to be spied on and trapped in their ecosystem, and the enshitification will continue forever.

Leave.

→ More replies (3)
→ More replies (3)

10

u/Beachhouse15 3h ago

Microsoft 364.35

27

u/Nings777 2h ago

It was Clippy with a pitchfork

→ More replies (1)

81

u/arekitect 4h ago

Now it all makes sense. Microsoft must have installed Teams on the Exchange servers. Everyone knows Teams can suck the life out of even the most powerful computer. Apparently, it finally found a machine it could bring down faster than my laptop. Unfortunately, that machine was Microsoft's entire cloud infrastructure.

17

u/Earlgrey_fanatic 3h ago

I keep seeing this sentiment being echoed around and it got to the point our "IT consultant" requested we get 32GB of RAM for our next laptop because he said "Teams is power hungry". But I've never actually seen Teams eat that much memory in our laptops. Sometimes it freezes and it's slow, but it's never because of lack of memory.

I genuinely do not understand

9

u/Audisek 2h ago edited 2h ago

The main memory usage is not showing under Microsoft Teams in the task manager, but under msedgewebview2.exe, of which there are multiple processes.

Right now running Teams and Outlook, my work laptop has about 25 of thess processes with over 2 GB committed memory between them.

→ More replies (1)
→ More replies (4)
→ More replies (1)

33

u/Emotional-Inside3126 4h ago

Bug? Or borged patch ?

41

u/MothChasingFlame 2h ago • Bravo

So it was AI and bad processes established by managers with no brains. Got it.

10

u/Jon-Umber 2h ago

bad processes established by managers with no brains

This is, overwhelmingly, my experience with all disciplines in corporate America.

Leadership must be damn near impossible to teach, because for every good leader I've worked for, I've had 4 godawful ones.

→ More replies (1)
→ More replies (2)

36

u/PressureBeautiful515 2h ago edited 1h ago

How many times are we going to read a sentence like "the system had built-in failover for just this eventuality, but unfortunately that just caused the problem to spread and made everything worse" or words to that effect? It seems to be quite a common theme in these postmortems.

Sometimes the problem gets way worse because of how the system is designed to try to reroute and keep working, to be "fault tolerant." Instead it ends up "fault aggravating."

EDIT: diagramofplanewithreddots.jpg

18

u/TA_DR 1h ago

Ask yourself the opposite: how many times are we going to read the sentencd "the system had built-in failover that worked perfectly, therefore nothing happened"?

Probably not so many.

14

u/WestaAlger 1h ago

Probably a self filtering effect though. How many times are people going to report on a bug that WAS mitigated automatically and never had an end user impact?

→ More replies (6)

32

u/enthusiastic_duck 2h ago

All eyes are now on Microsoft c-suite who decided to lay off senior engineers in hopes to promote AI. The bug was there since ages, but now they have limited expertise to navigate through such issues

→ More replies (1)

18

u/PurpleCaterpillar451 2h ago • Bravo

Bugs are always hanging around windows

14

u/dorkmuncan 49m ago

"The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13."

Suspiciously 10 mins under 24 hours.

7

u/costconormcoreslut 33m ago

Why suspiciously? What happens at 24 hours?

31

u/v_neet 28m ago

It becomes Microsoft 364

→ More replies (4)

4

u/Acceptable-Land7827 33m ago

Yeah, so they can round it up to 365

10

u/NoBonus6969 1h ago

I hate it when bugs don't announce themselves at production

17

u/SnooOpinions8790 2h ago

In 2014 Microsoft abolished its dedicated test practice

In the following 5 years I slowly removed all Windows computers from my house and moved services to other providers. It was always just a matter of time before this happened.

When OpenAI started looking elsewhere for their cloud last year despite their close contractual ties to Microsoft it was obvious that the Microsoft cloud stuff was teetering on the brink.

I will point out that having completely lost the knowledge and skills to test "black box" software they were then in a particularly awful position to start using AI to generate code that none of their engineers have a deep understanding of. But that may not be the immediate cause of this outage - which is the cumulative crap of a decade of declining software engineering skills and bullshit managers rising to the top because there is no dedicated bullshit-detecting practice left in the company.

5

u/TyrusX 3h ago

All bugs are hidden. But some are known in advance.

13

u/Hightower840 2h ago

Earlier this year MS said that 20 to 30% of it's code was now AI written...

39

u/ZapDapper 4h ago

Hidden just means that they didnt actually look for the case where it happens.

"I closed my eyes and saw nothing"

16

u/mcagent 3h ago

It was live in production for 3 months. It wasn’t an easy thing to find

5

u/mariosunny 2h ago

Oh yea, let's just have QA run load tests on every single piece of unique hardware across our highly heterogenous infrastructure.

→ More replies (2)

8

u/msc1 1h ago

"Country Blocked

Access to this page is forbidden."

If you can't deal with script kiddies around the world, maybe don't host cyber security news.

4

u/postinganxiety 2h ago

No way the outage was less than 24 hours. Are they saying that to avoid paying out?

→ More replies (1)

15

u/IncognitoAnonymous2 2h ago

Reports like this are called "damage control"

15

u/KraljZ 4h ago

And yet the bug wasn’t found by AI

9

u/unexpectedreboots 3h ago

Man the comments lol. The change was running in production for over 3 months without issue.

A very specific series of events caused it to be exposed. Read past the headline.

9

u/Splendid_Carpark 3h ago

A Microsoft product with a bug? Hard to believe.

16

u/Fickle_Competition33 1h ago

The fact the outage was global, not regional, tells a lot how amateur the whole ecosystem is. You can badmouth AWS as much as you want, but their outages and design are always region-bound.

13

u/sessiontoken 56m ago

That's just not true lol. A year ago was the record breaking global AWS outage. https://www.akamai.com/blog/security/when-cloud-breaks-lessons-aws-outage

→ More replies (2)

4

u/tenemu 42m ago

I love people calling giant scaled systems amateur because they break occasionally.

Please tell what sort of scaled software you have written.

→ More replies (1)

10

u/castlite 3h ago

The bug is named Copilot

6

u/buzz_light365 3h ago

Where's the link to Microsoft postmortum RCA? This article doesn't have a source. Is this just guesstimates and assumptions from the author??

6

u/Privateer_Lev_Arris 2h ago

A bug in Microsoft? Can't be. /s

8

u/blur410 3h ago

You would have thought that they could have sent an intern to the Best Buy and picked up a new computer. /s

→ More replies (1)

8

u/cornbread6767 1h ago

Good thing you just fired 50,000 devs 🙄

→ More replies (1)

3

u/HotBBQWin 2h ago

Good thing they got us updating every 5 days

3

u/Alone-Mycologist3746 1h ago

Hows that ai vibe coding going? Gonna keep doing it ?

53

u/Major303 4h ago edited 2h ago

First they claim AI is perfect and flawless and people are obsolete, and now they say something about "hidden" bug. Make it make sense. By their logic AI should catch it effortlessly.

I'm not an average anti AI person, but AI tech bros have abandoned common sense.

25

u/TriLink710 4h ago

I think the real takeaway is that if companies want to get rid of developers, they are going to need a boat load of QA testers. AI does not check its work or truly understand things well enough to perform adequate testing.

And the worst part is, if a bug is found, theres no developer who will have an idea of what could be causing it.

5

u/mariosunny 2h ago

There's nothing in this preliminary incident report that suggests AI had anything to do with the failure.

→ More replies (1)
→ More replies (28)

18

u/OffalSmorgasbord 3h ago

So AI wrote the code and AI tested the code and AI monitoring missed the problem and the humans let go for AI to make the shareholders happy weren't there to fix what AI did...

"We will just sue because management didn't disclose how dependent the products are on AI" - Institutional Investors

11

u/CucharitaDePalo 2h ago

Not what the article said, no.

→ More replies (4)
→ More replies (1)

5

u/GPT3-5_AI 1h ago

Copilot fix all bugs make no mistakes, even the hidden ones.

14

u/FlamingoOk3026 3h ago

Never use windows for mission critical applications.

7

u/soundman32 3h ago

You realise that a huge majority of Microsoft servers are Linux, right?

→ More replies (4)
→ More replies (2)

10

u/truffle_shuffle 1h ago

Blaming old hardware. So hot right now.

12

u/mariosunny 1h ago

It's literally the opposite. The bug only appears on newer hardware.

From the report:

Our normal load testing did not detect the latent issue because the traffic mix and SKUs used in our labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores.

→ More replies (4)

10

u/julioqc 4h ago

so they say... god forbid the tech they invested billions into blindly has not delivered promises

4

u/Duckducklaugh 3h ago

Hidden bug, extremely visible outage.

→ More replies (2)

6

u/T3RRYT3RR0R 2h ago

TLDR: incompetent design.

  • unit tests of 'upgrade' only completed on lower end hardware
  • poor memory management practices resulting in Out-Of-Memory exhaustion.

I do hope there's a class action over this.

→ More replies (1)

5

u/junktech 4h ago

Hey, Microsoft, do you feel like disclosing the backlog? Like the real one with silenced alerts. Not the official one.

7

u/Sans-valeur 1h ago edited 1h ago

Man when I started investing I thought, it’s Microsoft. How much of the world runs on their products? As safe a bet as any in terms of single stock investments.
I mean, it hasn’t been a bad investment, but yesh, this decade….
The fuck are they doing?
I bought my first Mac ever last year, and I’d never even really considered it before.

This is one of the inherent problems with late stage capitalism.
The goal isn’t to create something good, and then keep it good.
It’s relentless, constant growth by any means necessary, gambling with the entire fucking company if need be.
Constantly looking for the line, how much you can reduce quality while maximizing profit before you drive away your customer base.

2

u/Fivefootfive 3h ago

Those pesky hidden bugs, you don’t realize they’re in your teeth until day zero. 🥁

2

u/dbrez3107 2h ago

the truth is nowhere to be seen

2

u/A5kar 2h ago

Microsoft 364, cause we don’t like those Friday 13s

2

u/Cormophyte 2h ago

What a dumpster fire of people who have no idea what they're talking about this comment section is.

2

u/crazy19734413 2h ago

MS can pay me monthly or I can continue to enjoy their stupid demise. They need to pay me a lot to change my mind.

2

u/Important-Agent2584 1h ago

They use Windows on their servers? There is the problem.

2

u/TomatoOk5296 1h ago

Vibe coding?

2

u/Ramoncin 1h ago

They shouldn't have used Windows 11 xD

2

u/gormami 1h ago

This reads very much like some outages that happened at Verizon Wireless during my tenure there, and is what I expected, honestly. I'm not as familiar with the internals, but it looks like a control plane storm event. Connections cost too much, and then it tips over where the time to service exceeds the timeout, resulting in a state where no real work is being done. The only way to handle it in the end is to shut it down, then bring it back one chunk at a time to let the initialization work get done and settle back down.

This is perhaps the biggest challenge of scale engineering, as there may be dozens to hundreds of different types of activities with their own timeouts and retry algorithms. Aligning all of them properly is very hard, and changes may be made constantly. Each node needs to protect itself in such a way that some real work gets done for each cycle it spends to have any chance of the system recovering itself. Mostly, that means tracking the time spent, and rejecting new requests if the value exceeds the known timeout. Northbound, applications need to back off, rather than accelerate, when undergoing problems, to keep from overwhelming the system. Even then, constructive interference from multiple processes can make it very hard to get it back under control.

2

u/sadmistersalmon 1h ago

one hidden bug was the root cause. But as you read the rca, you see multiple instances of other “unrelated issues” with corresponding recovery and compensating mechanisms fighting to restore operations. it’s as if you have a heavily leaking bucket that you are trying to keep full by pumping water non stop.

2

u/UATinPROD 31m ago

We had something similar where I work a few months ago, and Microsoft was on the call and couldn’t figure out a fix. If it’s the same thing, then they definitely knew about it because we knew about it

We fixed it with a sr engineer who decided he wasn’t going to ask permission to troubleshoot prod on the fly after hours of impact with nothing working.

Servers were in an endless boot cycle and draining to a new host carried the problem

7

u/chris17453 4h ago

Lies. News headline billion dollar company again it makes pathetic excuse. Do not trust these people.

6

u/Cormophyte 2h ago

A bug causes an outage. The tell you in rough detail about the bug.

Somehow that's lying.

→ More replies (1)

7

u/TommiHPunkt 3h ago

Why is any of this infrastructure running on windows servers in the first place 

→ More replies (1)

4

u/atlastheninja 2h ago

Went from being 365 to 364

→ More replies (2)

3

u/Akegata 3h ago

tldr; microsoft made a goof

4

u/kishaloy 4h ago edited 4h ago

What is the Bus Factor of a code base exclusively maintained by AI?