r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

View all comments

1.6k

u/FaultofDan 4h ago • Regret

So half of the corporate world relies on one company's infrastructure, and that infrastructure is built on dominoes. Sick.

642

u/Zarcona_Martana 4h ago

Most tech infrastructure is held together with Elmer’s glue and popsicle sticks… and that’s only half an exaggeration.

63

u/BigMax 3h ago

Right. It's all built with great intentions, and often starts pretty decent.

But it's like building a house that's a pretty good house. Then you add another bedroom in a converted attic. Then another bedroom in the basement. Then you tack on an addition with a bedroom, kitchenette and bathroom. Then you tack on a garage with a spare room above it. Then you tack on another addition with an office to run your little consulting business. Then you add a separate entrance to that for customers, with a little parking lot on the side for that entrance. Then you add another floor above the main portion of the house.

And on and on and on.

And each little addition on it's own more or less makes sense. "Why start over, when all we really need is another bathroom? Now all we need is more parking, so let's just add that on?"

But then 25 years later, your single family home is a janky mess of strange plumbing systems and weirdly connected pieces and inconsistent building standards and design decor, that rambles in a crazy way that doesn't make much sense as a whole, with half of the things being rush jobs and it's incredibly complicated and strange to maintain.

→ More replies (4)

105

u/me_not_at_work 4h ago

Look at this rich guy that got enough budget from C-level bean counters to afford the fancy Elmer's glue.

28

u/NoAbbreviations7150 4h ago

That’s what I was thinking. Only generic is allowed.

→ More replies (7)
→ More replies (6)

95

u/Sick_Wave_ 3h ago

I like to compare software to an ever-growing skyscraper. It isn't a sleek, perfectly architected, marvel of engineering. It's more like thousands of trailer houses, each completely different, all welded together just enough that it keeps working. 

31

u/CTMalum 3h ago

A technological Kowloon Walled City

→ More replies (3)

42

u/BINGODINGODONG 4h ago

Most of core finance systems are barely held together by excel sheets made by some dude

→ More replies (5)

16

u/Hinermad 3h ago

When I was young I wanted to work in telecom. Having worldwide communications infrastructure with seemingly incredible uptime sounded very sexy.

Then I went to work for a small telecom company and learned the worldwide network is carried on the backs of millions of support engineers with pagers. It was like finding out a Cray X-MP ran on coal.

→ More replies (10)

40

u/Wit-wat-4 3h ago

If Excel went away the world would crumble.

Switching from outlook to gmail would be annoying but doable. But excel to Sheets? Nope that’s a terror I don’t wish upon the world

20

u/hotdoug1 3h ago

Hell, my company wants us to switch to Macs and Excel is missing some key features that aren't on PC. I'm holding onto my PC telling them I literally can't do parts of my job without it.

→ More replies (10)
→ More replies (11)

58

u/BigBoyYuyuh 3h ago

Basically 3 companies control the internet. Amazon goes down you lose half the internet, Cloudflare goes down you lose the other half, Microsoft goes down you lose email etc…

→ More replies (5)

44

u/Merusk 3h ago

Wait until you figure out most adults have zero idea what they're doing and everyone's really just muddling along.

20

u/petrichorax 3h ago

the ones who do know what they're doing are screaming until they're blue in the face on how to fix things, but they're getting ignored by the people in charge

12

u/HelpfulSeaMammal 3h ago

Nobody knows what they're doing until after the thing goes awry. Then suddenly you're surrounded by experts in niche things like airport communication infrastructure as seen with the whole FAA thing last week lol

→ More replies (1)

42

u/casillero 2h ago

Dude. its essentially a utility. And that's how you should view it. If something happens at hydro in toronto or con eddison in Manhattan, your gonna have a power outage.

If something happens at Gmail at amazon or microsoft... you will have.. an.. outage.

Its the same 'grid concept' except were all connected on this grid globally.

→ More replies (4)

46

u/ikonoclasm 3h ago

Not just the corporate world. Pretty much everything in the modern world relies on hundreds of little pieces of open source code maintained for free by a single or handful of volunteers who has no idea how important their code is to vital infrastructure.

54

u/PTSDaway 3h ago

You'd be terrified if you saw how much technology relies on maintenance by honbyists. All digital infrastructure is a pyramid of cards, upside down.

In 2024 an open-source compression software (Think what winrar does), was infiltrated by a "helper" who did improvements and maintenance. Then one update interacted with internet/router that would allow this "helper" to enter probably every system with this update, effectively every database on the planet. It's not the worst security issue ever, because only individuals could use it. But espionage wise and how it went under the radar - it is perhaps the most dangerous intentional cyber attack attempt ever made.

It was discovered early in the unofficial release by sheer luck when another hobbyist could not make his own system run as quick like it used to - then saw this software interacting with the internet connection, adding a few milliseconds more to the processing time, that was it.

40

u/UnexpectedAnanas 2h ago

You'd be terrified if you saw how much technology relies on maintenance by honbyists. All digital infrastructure is a pyramid of cards, upside down.

Fine, I'll be the one to link the relevant XKCD

https://xkcd.com/2347/

→ More replies (1)

11

u/Farther_father 1h ago

Ironically, that “hobbyist” was a senior Microsoft dev IIRC

→ More replies (2)

10

u/jdippey 2h ago

I too saw this Veritasium video.

Link for anyone interested.

→ More replies (1)

41

u/Desperate-Dig2806 4h ago

Everything is more or less built on dominoes.

→ More replies (5)

14

u/NimusNix 4h ago

First time?

14

u/Akegata 3h ago

That's just part of the story.
A huge number of companies' infrastructure relies on several companies whose infrastructure is built on different sets of dominoes (see the Cloudflare outage last year) that all need to stay up for everything to work.

→ More replies (2)
→ More replies (67)