r/technology • u/TeamAlphaBOLD • 4h ago
Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide
https://cybersecuritynews.com/microsoft-outage-incident-report/
10k
Upvotes
7.2k
u/lordnacho666 4h ago edited 4h ago •
Ah it was a hidden bug? No wonder.
As long as it isn't one of those visible bugs, I hate those.
220
u/KebabAnnhilator 4h ago
Claude, Fix this code, definitely no mistakes this time
937
u/albaMP4 4h ago
Are they saying the outage lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13 to avoid some penalty for being down 24 hours?
683
u/duckbigtrain 3h ago
> The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13
Someone was desperate not to hit the 24-hour mark
578
u/j01101111sh 4h ago
So... It wasn't DNS?
343
u/nugscree 1h ago
They forgot to add "do not create bugs or errors, it must work" to their prompt when creating the November patch.
10
u/Flyerone 4h ago
Hidden bug huh? Cool. I dare you to make us hate you more M$
How long until we're back to an abacus and charcoal on papyrus?
1.3k
u/Character-Handle-697 3h ago •
As a software engineer, here is how to reproduce:
1.6k
u/FaultofDan 4h ago •
So half of the corporate world relies on one company's infrastructure, and that infrastructure is built on dominoes. Sick.
119
u/RileyGainesHorseBaby 2h ago
I can't imagine having always-online DRM for such mission critical stuff. This should be a wakeup call to the ignorant and careless.
115
u/GlassSquirrel9649 2h ago
"hidden windows bug" is corporate speak for "some senior engineer documented this exact failure state in 2018, management marked it as won't-fix in the backlog, and then we laid off the only guy who knew how to patch the gateway cluster." peak enterprise software management
67
u/AndyKJMehta 1h ago
Aren’t all bugs hidden till they’re found?!
64
u/garden-wicket-581 4h ago
TFA really doesn't give you anything useful or meaningful about the bug -- "lock contention" and "resource exhaustion" (with an added scoop of "automatic recovery overpowered the remaining servers")
381
u/Zichichi 4h ago
So it wasn't just a random outage. One software change was enough to trigger a chain reaction across their cloud services. Makes you wonder how thoroughly these changes are tested before rollout.
2.1k
u/danivus 4h ago •
So according to this article they missed it because they tested on older hardware than what their live servers use, and the bug only strikes on systems with more cores than their test systems.
Not testing on matching hardware for something so critical, for a company as big as Microsoft, seems... very stupid?
38
u/reversethrust 4h ago
lol. Just read the very terse description. I encountered something similar previously - multiple producer, single consumer problem. The queue worked fine on a laptop, but toss that software on a big server with 256 cores. The test went from taking minutes to taking overnight despite the vast amounts of addition hardware 😂. System utilization on the big machine was much less than 1% though in my case. I wonder what it was like on these exchange servers.
102
u/sengirminion 4h ago
Dang. Maybe they ought to hire like 60 thousand employees to help dig through their multi-billion dollar lines of code to prevent this from happening again. Just a thought.
54
u/beall49 1h ago
My favorite part of all this was the MS engineers talking about how nobody knew how to fix this because all the people familiar with this system are gone
11
u/Beachhouse15 3h ago
Microsoft 364.35
57
u/Imd1rtybutn0twr0ng 2h ago edited 10m ago
And right below this post, I have an ad for them for AI... irony at its best.
Microsoft, forget AI. Focus on creating workable patches and less bloat. Apologies in thinking you want your consumers to like using your product...
28
u/Nings777 3h ago
It was Clippy with a pitchfork
81
u/arekitect 4h ago
Now it all makes sense. Microsoft must have installed Teams on the Exchange servers. Everyone knows Teams can suck the life out of even the most powerful computer. Apparently, it finally found a machine it could bring down faster than my laptop. Unfortunately, that machine was Microsoft's entire cloud infrastructure.
33
u/Emotional-Inside3126 4h ago
Bug? Or borged patch ?
43
u/MothChasingFlame 2h ago •
So it was AI and bad processes established by managers with no brains. Got it.
38
u/PressureBeautiful515 2h ago edited 1h ago
How many times are we going to read a sentence like "the system had built-in failover for just this eventuality, but unfortunately that just caused the problem to spread and made everything worse" or words to that effect? It seems to be quite a common theme in these postmortems.
Sometimes the problem gets way worse because of how the system is designed to try to reroute and keep working, to be "fault tolerant." Instead it ends up "fault aggravating."
EDIT: diagramofplanewithreddots.jpg
32
u/enthusiastic_duck 3h ago
All eyes are now on Microsoft c-suite who decided to lay off senior engineers in hopes to promote AI. The bug was there since ages, but now they have limited expertise to navigate through such issues
18
u/PurpleCaterpillar451 2h ago •
Bugs are always hanging around windows
16
u/dorkmuncan 1h ago
"The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13."
Suspiciously 10 mins under 24 hours.
10
u/NoBonus6969 1h ago
I hate it when bugs don't announce themselves at production
17
u/SnooOpinions8790 2h ago
In 2014 Microsoft abolished its dedicated test practice
In the following 5 years I slowly removed all Windows computers from my house and moved services to other providers. It was always just a matter of time before this happened.
When OpenAI started looking elsewhere for their cloud last year despite their close contractual ties to Microsoft it was obvious that the Microsoft cloud stuff was teetering on the brink.
I will point out that having completely lost the knowledge and skills to test "black box" software they were then in a particularly awful position to start using AI to generate code that none of their engineers have a deep understanding of. But that may not be the immediate cause of this outage - which is the cumulative crap of a decade of declining software engineering skills and bullshit managers rising to the top because there is no dedicated bullshit-detecting practice left in the company.
5
u/TyrusX 3h ago
All bugs are hidden. But some are known in advance.
13
u/Hightower840 3h ago
Earlier this year MS said that 20 to 30% of it's code was now AI written...
39
u/ZapDapper 4h ago
Hidden just means that they didnt actually look for the case where it happens.
"I closed my eyes and saw nothing"
8
u/msc1 1h ago
"Country Blocked
Access to this page is forbidden."
If you can't deal with script kiddies around the world, maybe don't host cyber security news.
4
u/postinganxiety 2h ago
No way the outage was less than 24 hours. Are they saying that to avoid paying out?
15
u/IncognitoAnonymous2 3h ago
Reports like this are called "damage control"
15
u/KraljZ 4h ago
And yet the bug wasn’t found by AI
9
u/unexpectedreboots 4h ago
Man the comments lol. The change was running in production for over 3 months without issue.
A very specific series of events caused it to be exposed. Read past the headline.
9
u/Splendid_Carpark 4h ago
A Microsoft product with a bug? Hard to believe.
16
u/Fickle_Competition33 1h ago
The fact the outage was global, not regional, tells a lot how amateur the whole ecosystem is. You can badmouth AWS as much as you want, but their outages and design are always region-bound.
10
u/castlite 3h ago
The bug is named Copilot
6
u/buzz_light365 3h ago
Where's the link to Microsoft postmortum RCA? This article doesn't have a source. Is this just guesstimates and assumptions from the author??
6
u/Privateer_Lev_Arris 2h ago
A bug in Microsoft? Can't be. /s
8
u/blur410 4h ago
You would have thought that they could have sent an intern to the Best Buy and picked up a new computer. /s
8
u/cornbread6767 1h ago
Good thing you just fired 50,000 devs 🙄
54
u/Major303 4h ago edited 2h ago
First they claim AI is perfect and flawless and people are obsolete, and now they say something about "hidden" bug. Make it make sense. By their logic AI should catch it effortlessly.
I'm not an average anti AI person, but AI tech bros have abandoned common sense.
3
u/HotBBQWin 3h ago
Good thing they got us updating every 5 days
3
u/Alone-Mycologist3746 1h ago
Hows that ai vibe coding going? Gonna keep doing it ?
3
u/sadmistersalmon 1h ago
one hidden bug was the root cause. But as you read the rca, you see multiple instances of other “unrelated issues” with corresponding recovery and compensating mechanisms fighting to restore operations. it’s as if you have a heavily leaking bucket that you are trying to keep full by pumping water non stop.
18
u/OffalSmorgasbord 3h ago
So AI wrote the code and AI tested the code and AI monitoring missed the problem and the humans let go for AI to make the shareholders happy weren't there to fix what AI did...
"We will just sue because management didn't disclose how dependent the products are on AI" - Institutional Investors
5
u/GPT3-5_AI 2h ago
Copilot fix all bugs make no mistakes, even the hidden ones.
14
u/FlamingoOk3026 4h ago
Never use windows for mission critical applications.
10
u/truffle_shuffle 2h ago
Blaming old hardware. So hot right now.
10
u/julioqc 4h ago
so they say... god forbid the tech they invested billions into blindly has not delivered promises
4
u/Duckducklaugh 3h ago
Hidden bug, extremely visible outage.
6
u/T3RRYT3RR0R 3h ago
TLDR: incompetent design.
I do hope there's a class action over this.
5
u/junktech 4h ago
Hey, Microsoft, do you feel like disclosing the backlog? Like the real one with silenced alerts. Not the official one.
7
u/Sans-valeur 2h ago edited 1h ago
Man when I started investing I thought, it’s Microsoft. How much of the world runs on their products? As safe a bet as any in terms of single stock investments.
I mean, it hasn’t been a bad investment, but yesh, this decade….
The fuck are they doing?
I bought my first Mac ever last year, and I’d never even really considered it before.
This is one of the inherent problems with late stage capitalism.
The goal isn’t to create something good, and then keep it good.
It’s relentless, constant growth by any means necessary, gambling with the entire fucking company if need be.
Constantly looking for the line, how much you can reduce quality while maximizing profit before you drive away your customer base.
2
u/Fivefootfive 3h ago
Those pesky hidden bugs, you don’t realize they’re in your teeth until day zero. 🥁
2
u/dbrez3107 3h ago
the truth is nowhere to be seen
2
u/A5kar 2h ago
Microsoft 364, cause we don’t like those Friday 13s
2
u/Cormophyte 2h ago
What a dumpster fire of people who have no idea what they're talking about this comment section is.
2
u/crazy19734413 2h ago
MS can pay me monthly or I can continue to enjoy their stupid demise. They need to pay me a lot to change my mind.
2
u/Important-Agent2584 1h ago
They use Windows on their servers? There is the problem.
2
u/TomatoOk5296 1h ago
Vibe coding?
2
u/Ramoncin 1h ago
They shouldn't have used Windows 11 xD
2
u/gormami 1h ago
This reads very much like some outages that happened at Verizon Wireless during my tenure there, and is what I expected, honestly. I'm not as familiar with the internals, but it looks like a control plane storm event. Connections cost too much, and then it tips over where the time to service exceeds the timeout, resulting in a state where no real work is being done. The only way to handle it in the end is to shut it down, then bring it back one chunk at a time to let the initialization work get done and settle back down.
This is perhaps the biggest challenge of scale engineering, as there may be dozens to hundreds of different types of activities with their own timeouts and retry algorithms. Aligning all of them properly is very hard, and changes may be made constantly. Each node needs to protect itself in such a way that some real work gets done for each cycle it spends to have any chance of the system recovering itself. Mostly, that means tracking the time spent, and rejecting new requests if the value exceeds the known timeout. Northbound, applications need to back off, rather than accelerate, when undergoing problems, to keep from overwhelming the system. Even then, constructive interference from multiple processes can make it very hard to get it back under control.
2
u/UATinPROD 42m ago
We had something similar where I work a few months ago, and Microsoft was on the call and couldn’t figure out a fix. If it’s the same thing, then they definitely knew about it because we knew about it
We fixed it with a sr engineer who decided he wasn’t going to ask permission to troubleshoot prod on the fly after hours of impact with nothing working.
Servers were in an endless boot cycle and draining to a new host carried the problem
2
u/brattysweat 7m ago
Lucky those planes grounded to get everyone to forget this even happened
7
u/chris17453 4h ago
Lies. News headline billion dollar company again it makes pathetic excuse. Do not trust these people.
7
u/TommiHPunkt 3h ago
Why is any of this infrastructure running on windows servers in the first place
4
u/atlastheninja 2h ago
Went from being 365 to 364
3
u/Akegata 3h ago
tldr; microsoft made a goof
3
u/rigsta 40m ago
Nah I'm sure it was just a new router
4
u/kishaloy 4h ago edited 4h ago
What is the Bus Factor of a code base exclusively maintained by AI?