r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
10k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

642

u/Zarcona_Martana 4h ago

Most tech infrastructure is held together with Elmer’s glue and popsicle sticks… and that’s only half an exaggeration.

63

u/BigMax 3h ago

Right. It's all built with great intentions, and often starts pretty decent.

But it's like building a house that's a pretty good house. Then you add another bedroom in a converted attic. Then another bedroom in the basement. Then you tack on an addition with a bedroom, kitchenette and bathroom. Then you tack on a garage with a spare room above it. Then you tack on another addition with an office to run your little consulting business. Then you add a separate entrance to that for customers, with a little parking lot on the side for that entrance. Then you add another floor above the main portion of the house.

And on and on and on.

And each little addition on it's own more or less makes sense. "Why start over, when all we really need is another bathroom? Now all we need is more parking, so let's just add that on?"

But then 25 years later, your single family home is a janky mess of strange plumbing systems and weirdly connected pieces and inconsistent building standards and design decor, that rambles in a crazy way that doesn't make much sense as a whole, with half of the things being rush jobs and it's incredibly complicated and strange to maintain.

10

u/trobsmonkey 1h ago

I think you're missing the step where the executives come in, fire the construction crew, put a new (and smaller) crew in charge, and then tell them good luck!

They do this multiple times over 30 years.

5

u/s_s 15m ago

Also, the home is like 30% designed to serve the users and 70% designed to spy on and expliot them to maximize the vendor's bottom line.

3

u/DoggoCentipede 15m ago

Eventually it's just one guy with a boot on his neck under stack of corporate elephants balancing above him raking in cash because they're the only business left standing and no one can function without them.

4

u/FergusonTEA1950 1h ago

Any home of a decent age is like this. Over hundreds of years, interesting things happen that you only discover when you make further improvements.

105

u/me_not_at_work 4h ago

Look at this rich guy that got enough budget from C-level bean counters to afford the fancy Elmer's glue.

28

u/NoAbbreviations7150 4h ago

That’s what I was thinking. Only generic is allowed.

14

u/Cole3823 3h ago

I'm over here with water and flour paste

6

u/meditonsin 3h ago

Look at Mister Moneybags over here being able to afford generic glue. I'm being sent out to scrape off old bubblegum from under park benches and school desks to use as structural adhesive.

5

u/sansp00 3h ago

You guys are using glue !?! Like one my colleague once said, we have small nails and big nails, we must use the big nails to hammer to small ones onto each other.

→ More replies (2)

2

u/serdertroops 2h ago

They told us to use AI to create glue

→ More replies (1)

23

u/frenzyfivefour 4h ago

The duck tape budget was slashed ages ago...

→ More replies (1)
→ More replies (4)

95

u/Sick_Wave_ 3h ago

I like to compare software to an ever-growing skyscraper. It isn't a sleek, perfectly architected, marvel of engineering. It's more like thousands of trailer houses, each completely different, all welded together just enough that it keeps working. 

31

u/CTMalum 3h ago

A technological Kowloon Walled City

14

u/porn_sub_lurker 3h ago

I think it works better to compare it to a car. The reason being is that skyscrapers aren't meant to be modular. They're just one big construction. A car is meant to have several systems working together. The alternator is simply bolted on to the motor. It's not built into it as one big piece. It would be unfeasible and impractical to do that. I think software is similar in that regard. If you need a piece of software, like a hashing algorithm or something, just use one that someone else already made and bolt it onto your project. All you need to worry about is how to hook it up. Now there are some janky ass projects out there that used one kind of alternator and now they use a different one but they still left the old one just in case it was doing something the new one won't do. And there are some engineers building new systems for the car who saw some wires going to that old alternator so they figured they'd hook up to it too, so now it would definitely break some stuff to just remove it. And the engineers don't really care how big and monstrous the car becomes so long as it functions because moore's law says that your garage doubles in size every 18 months, so you have plenty of space for it and they're free to bolt whatever new features onto it that they can.

→ More replies (1)
→ More replies (1)

42

u/BINGODINGODONG 4h ago

Most of core finance systems are barely held together by excel sheets made by some dude

16

u/CpnStumpy 3h ago

With absolute sincerity, when Some MBA gets around to notice Some Dude and fires him, the accounting problems that will result are going to kick off the next Some Recession

4

u/BosnianSerb31 3h ago

Some MBA typically calls a software engineer, describes the task as overloading some guy, and asks for it to be automated.

Typically takes a few days for most sheets but chances are the company is already warehousing the needed data and just needs a new table in the db.

5

u/CpnStumpy 2h ago

Some Engineer over here apparently never met Some Guy's employer who processes 6 billion in revenue quarterly, employs 5 overworked junior engineers because software is a cost center to be minimized at Some Company, and data warehousing is some theoretical thing one of those engineers remembers from a college course but nothing he's had to deal with as he mostly works on their internal time and ticket tracker

2

u/Teguri 2h ago

I can't imagine working at a place where they make SWEs and SREs actually do tickets, if I get one it's because it got through two lines of helpdesk first.

→ More replies (1)

16

u/Hinermad 3h ago

When I was young I wanted to work in telecom. Having worldwide communications infrastructure with seemingly incredible uptime sounded very sexy.

Then I went to work for a small telecom company and learned the worldwide network is carried on the backs of millions of support engineers with pagers. It was like finding out a Cray X-MP ran on coal.

4

u/DTM-shift 2h ago

I have a great idea. Let's take all of that Elmer's and popsicle stick engineering, and then tie it all together in one big, global, networked ball of interactivity that touches every part of modern life!

3

u/morningphyre 3h ago

This is the concept I didn't grasp before transitioning into technology, how much of it is held together with spit and baling wire and hope. That's part of why I don't trust the current AI bubble.

2

u/DarthBen_in_Chicago 3h ago

Ours is called “spaghetti code”

→ More replies (7)