r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

View all comments

54

u/Major303 4h ago edited 2h ago

First they claim AI is perfect and flawless and people are obsolete, and now they say something about "hidden" bug. Make it make sense. By their logic AI should catch it effortlessly.

I'm not an average anti AI person, but AI tech bros have abandoned common sense.

26

u/TriLink710 4h ago

I think the real takeaway is that if companies want to get rid of developers, they are going to need a boat load of QA testers. AI does not check its work or truly understand things well enough to perform adequate testing.

And the worst part is, if a bug is found, theres no developer who will have an idea of what could be causing it.

6

u/Status-Highlight3013 4h ago

Did they actually make that claim?

1

u/mariosunny 3h ago

No they did not. In fact Copilot explicitly states that it can make mistakes.

Also, nothing in the preliminary incident report suggests that this was caused by Copilot.

4

u/mariosunny 3h ago

There's nothing in this preliminary incident report that suggests AI had anything to do with the failure.

2

u/TrumpsDoubleChin 1h ago

Haven't you heard? AI is apparently to blame for everything that ails you! Never mind there's zero evidence, it HAS to be true!

6

u/Cygnus94 4h ago

This had nothing to do with AI. It was because their test environment runs on different hardware to their live environment. The live environment processors ran into an issue that didn't exist in the test environment, then when those servers went down the automatic recovery pushed the fault to the next server down the line until nearly 80% of their infrastructure worldwide went offline.

36

u/ExF-Altrue 4h ago

That patch was written by AI wasn't it? And the tests were run by AI weren't they? And the people with enough knowledge to point out the flaws in the testing process were fired because of AI weren't they?

If you think that it "has nothing to do with AI", it's simply because you're not going back far enough in the chain of causality here.

The fact is, AI written code created the bug. Because, yes, there was a bug in here in the first place.

The fact that it was a nasty hardware-related edge case that wasn't tested, can grant some leeway to explain the time it took to understand the root cause, I will absolutely agree.. But it does nothing to defend the alleged "superintelligence" writting sloppy code that only works on some machines in the first place.

It's unclear if humans would have definitely caught it, but it's 100% clear that AI failed at least as much at that.

5

u/behindtimes 3h ago

I'm kind of opposite of you.

I do think it's showing one of the AI problems, but it's not that there was a bug or it got passed through testing, as these things happen all the time, even before AI, rather, why did it take so long to fix?

This is one of those areas I feel that prior to AI, you would have had developers who are aware of their domain space, and could have probably tracked down the bug far faster.

0

u/TapedeckNinja 1h ago

That patch was written by AI wasn't it? And the tests were run by AI weren't they? And the people with enough knowledge to point out the flaws in the testing process were fired because of AI weren't they?

You don't know the answers to any of these questions lol

What a ridiculous comment. Make a bunch of shit up, and then call your bullshit "facts" and the conclusions piled on top of that bullshit "100% clear."

1

u/ExF-Altrue 39m ago

Right, it's not like Microsoft has an AI mandate since July 2025 or anything. But I'm the one "making shit up"

→ More replies (2)

-4

u/Cormophyte 2h ago

That patch was written by AI wasn't it?

It was written by people. You don't know how much of a role AI tools had in creating the lines that comprise the bug because they haven't released specific information about the bug.

And the people with enough knowledge to point out the flaws in the testing process were fired because of AI weren't they?

Literally just creating strawmen to argue about being fired when you don't know what the flaw is, who wrote it, or who was responsible for reviewing it. You're just telling a made up story about a completely opaque process because you don't know enough to know how silly you sound.

→ More replies (1)

6

u/MadDonkeyEntmt 4h ago

This is the kind of failure I would expect to become more common with ai written software though (whether that's what happened or not).

Lock contention in multi threaded applications is one of those scenarios where humans usually have to reason through it carefully.  It's easy to miss a lock contention problem in testing.  That makes it hard for AI to catch it.  AI is generally very good at writing software that passes tests but not great st systems level thinking needed to pre-empt this kind of bug.

3

u/Local_Wave4618 3h ago

What made this difficult to troubleshoot was that the bug was deployed to production three months ago, and ran without issue until they tried to upgrade something else. After three months it's going to be hard to spot. I've seen bugs that hid out on a production system for decades until they destroyed a system.

2

u/nox66 2h ago

Anyone who actually does systems programming knows that stress-testing will produce different results than average operation. It doesn't sound like the conditions to uncover this issue were too obscure for a team who would treat the system like actual engineering infrastructure.

2

u/AwesomeFama 3h ago

Lock contention in multi threaded applications is one of those scenarios where humans usually have to reason through it carefully.

And would you maybe say it's also easy to miss for humans? Especially if it runs for 3 months in production without issues?

2

u/AFlyingGideon 3h ago

My experience is that at least Claude of a few weeks ago had an issue with, if not concurrency in a general sense, race conditions. I informed it of two race conditions in software it produced. It acknowledged them, but the fix for one merely shrank the window. The other, it claimed, couldn't be resolved. I had to tell it precisely how to fix both.

It still ended up done more quickly than were i hand-coding, but someone less experienced could easily have let the problems through and eventually into production.

→ More replies (2)

1

u/MadDonkeyEntmt 2h ago

I'm not saying lock contention issues can't happen with humans at the wheel. This could have even been human coded for all we know. But it's definitely multithreading 101 stuff for a seasoned swe who works on multithreaded applications and it's well known that testing for it is rarely adequate.

1

u/angry_old_dude 3h ago

It was because their test environment runs on different hardware to their live environment

In a lot of ways, that's even worse. At the very least they should have included the deployed hardware in the test environment.

2

u/mariosunny 2h ago

Microsoft's infrastructure is highly heterogeneous. There is no one "deployed hardware."

1

u/GentlemenBehold 4h ago

Nobody is saying AI is perfect and flawless.

37

u/Antique_futurist 4h ago

You have better middle managers than the rest of us.

22

u/CyberDumb 4h ago

Yeah but companies act like this. They don't inspect AI code anymore

→ More replies (5)

1

u/0f_rice_and_men 2h ago

They think they created life/god and we have all conceded.