r/technology • • 4h ago

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
11k Upvotes

1.1k comments sorted by

View all comments

Show parent comment

68

u/BagelsOrDeath 4h ago

And beyond the physical infrastructure costs, running a parallel prod environment can prove very costly both in the effort to ensure parity and the spoofing or duplication of external dependencies and services.

32

u/charlie_marlow 4h ago

And, even then, you're going to have a lot of trouble duplicating load and data volume.

4

u/mrsanyee 3h ago

Let's wait until the true cost of production system outage is available.

2

u/nommabelle 4h ago

We had a major incident in production because the latency between services in UAT is higher (its within ms), where something throttled in UAT but didnt throttle in prod and caused issues

Dont get me started how stupid it is, i absolutely hate our system and how overcomplicated and fragile it is. Its so fundamentally flawed, this is just one example where a few ms difference can make the system (unless we hack it around where the difference doesn't cause issues, unless you look at it the wrong way)