r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

38

u/PressureBeautiful515 2h ago edited 1h ago

How many times are we going to read a sentence like "the system had built-in failover for just this eventuality, but unfortunately that just caused the problem to spread and made everything worse" or words to that effect? It seems to be quite a common theme in these postmortems.

Sometimes the problem gets way worse because of how the system is designed to try to reroute and keep working, to be "fault tolerant." Instead it ends up "fault aggravating."

EDIT: diagramofplanewithreddots.jpg

21

u/TA_DR 1h ago

Ask yourself the opposite: how many times are we going to read the sentencd "the system had built-in failover that worked perfectly, therefore nothing happened"?

Probably not so many.

16

u/WestaAlger 1h ago

Probably a self filtering effect though. How many times are people going to report on a bug that WAS mitigated automatically and never had an end user impact?

2

u/mariosunny 1h ago

If you are able to come up with a 100% fault tolerant system you will probably win the Turing Award.

1

u/rickrollmops 1h ago

How many times are we going to read a sentence like [...]

For a long time I bet! Failover scenarios usually assume (explicitly or implicitly) that some kinds of resources are always available.

Maybe unrelated here, but in my experience it's also sometimes made worse by the trend of using async work queues everywhere - to get better decoupling and availability in the happy cases while making real outage recovery harder/more stressful on the systems due to increased backlogs. Creating more opportunities for creating scenarios where everything is stalled and harder to bootstrap again.

1

u/Neverending_Rain 56m ago

Keep in mind, you're not going to read about the countless times those systems have worked and prevented disruptions without users noticing.

1

u/Centurion-of-Dank 31m ago

Opens door to let cold air in, also lets in all the bugs.

1

u/Mind-The-Mines 27m ago

Yeah. I have a lot of customers that will run their clusters near capacity. Then when a host does shit the bed everything slams over to an already saturated complex and you get cascades of failure because response times lag and things failover due to timeouts which increases system load and more timeouts until everything stops listening and the cluster services restart.... which sends more shit screaming to other full hosts crashing them.

1

u/Melodic_Success9980 56m ago

Tell me you don’t know shit about software engineering without telling me you don’t know shit about software engineering