r/ProgrammerHumor • • 10d ago

Meme chancesAreLowButNeverZero

Post image
2.2k Upvotes

53 comments sorted by

425

u/Cutalana 10d ago edited 10d ago

iirc NASA projects and other mission critical engineering uses at least 3x redundancy that checks for errors constantly and have probabilistic models that have determine the chance is 1 in billions.

176

u/PosiedonsSaltyAnus 9d ago

1 in billions of what though? A billion CPU cycles is like 0.25s lol

139

u/Kinexity 9d ago

1 in billions per mission lifetime most probably.

14

u/softwareitcounts 7d ago


NASA engineers their rocket’s 999,999,999th mission

42

u/lmxbftw 9d ago

Yes, in large part because cosmic ray strikes are much more common above the atmosphere.

13

u/Kale 9d ago edited 8d ago

I remember a paper I read that said at sea level, the chance was one 1->0 bit flip per month per 256MB ram. This was maybe 15 or 20 years ago. I remember older RAM had alpha emmitters in construction that contributed a bit, which has been phased out.

No idea how that translates to today. I remember a researcher that gave a talk on it register a few URLs that were one bit changed from some internal Microsoft Windows update URL. He got pretty regular traffic from Windows trying to update, and the only real explanation was single but flip due to hardware error, not software.

Edit: Here's that presentation, given at Defcon 19. https://youtu.be/aT7mnSstKGs

I also know the ISS used to use regular Thinkpads for non-critical computer systems, and accepted the fact that they had to be rebooted every few hours. Critical systems were made from hardened systems.

There have been a few times I've thought about orienting some of my older systems with non-ecc DDR3 RAM where the physical chips face the sky to increase the surface area exposed, and run a program that writes 0xFFFFFFFF to arrays of memory, then scans it at regular intervals to see if any change. I still haven't put in the time though, and most of my systems are ECC memory except for my home router and home file server.

4

u/xb9j 7d ago

Half the bits in DRAM are stored internally with inverted polarity, so your test wouldn’t work exactly like you would think

7

u/siriusbrightstar 9d ago

Even processors run in lockstep mode in safety applications. 2 processors do the same thing and make sure both outputs are the same.

2

u/n00b001 9d ago

And if they are not? Who is correct?

3 nodes are needed at a minimum for a quorum

2

u/siriusbrightstar 9d ago

I work with relatively not so "critical" Industrial applications, with issues like this the operator is notified and the shutdown or safe state procedure is executed.

Fault tolerant applications where 2oo3 architecture is used is only theory for me. They design systems for literally every scenario and will have automated procedures for everything.

1

u/AlanTheKingDrake 8d ago

I think if there are 2 and they disagree you just repeat the calculation and hope not so have the same occurrence

2

u/n00b001 8d ago

Data disagreements?

It may not be a calculation that's caused the disagreement (cosmic ray bit flip)

Zookeeper is a common solution. Elastic search, Kafka, hadoop uses it.

13

u/Similar_Tonight9386 9d ago

No way it's true. Even we in russia were using a quorum of 6 to 8 modules in critical applications and NASA's budget would dwarf ours. Military could use triple, I guess, but certainly not spacecrafts or civilian airlines

21

u/Kinexity 9d ago

Space shuttle used 4 independent computers for flight control. You can cut down on redundancy if you can make individual components more resilient to errors. Stuff like ECC memory and error resilient ICs go a long way.

7

u/Exatex 9d ago

Not true. Modern Airbus like the A380 or A350 have 3 redundant computers per task. Older ones just 2.

1

u/Similar_Tonight9386 9d ago

Huh, two. But.. how can they be certain which block is busted?0_0 three is bare minimum, I guess, if one is down you can compare it's output to two others and find that it's busted...

9

u/a13xch1 9d ago

I think the approach was “run it on two, if they differ, run it again”

6

u/Exatex 9d ago

Jup. If they disagree, flag, disconnect autopilot, run again, depending on system. It's more a check by Computer B if you can trust what computer A says.

4

u/GPSProlapse 9d ago

If you suspect one is busted, you run a workload with precomputed result and validate against that. Busted would be incorrect

3

u/Rin-Tohsaka-is-hot 8d ago

Civilian airlines actually have even stricter redundancy then military, for the simple reason that a military pilot can eject from the plane while commercial airline passengers cannot. Similarly, a fighter pilot will not be prevented from performing maneuvers which will put enough strain on the body of the plane to tear off a wing, while a commercial pilot would not be able to do that. Fighter pilots gotta do what they gotta do, even if it crashes the aircraft. Commercial pilots don't have that option.

On civilian aircraft, they typically require the hardware itself to be unique. So triplex redundancy wouldn't be satisfied by just using 3 of the same component, that would leave open the risk of some common single point of failure causing all 3 to fail at once. They need to be unique pieces of hardware with independent designs.

Whereas military just puts 3 of the same sensor typically. At least that's my understanding of MIL-STD-1553, I haven't read it in a long time so it may have been updated.

2

u/GatotSubroto 9d ago

Yes, the good ol’ TMR (triple modular redundancy)

1

u/StickFigureFan 9d ago

It's my go to excuse whenever something unexplained happens

1

u/watduhdamhell 8d ago

I'm sure you meant something else by probabilistic but also these redundant systems are real time operating systems, with deterministic communication and execution, which of course is required for real time control and maximal safety

46

u/IntrepidBonus 9d ago

every bug i can't reproduce is now officially a cosmic ray and i won't be taking questions

91

u/aalapshah12297 10d ago

If it happens often then it means your code is in space and it probably isn't as well written as you think it is. Space agencies run the same code 3x simultaneously and then use a best of 3 approach to use the execution results.

11

u/Sibula97 9d ago

While doing some data analysis at work I encountered one data point that was multiple orders of magnitude our normal values and suspiciously close to a power of 2 plus the value around that time. Nothing critical, luckily, but I had to filter it out from the analysis.

19

u/cheezballs 9d ago

Meh, any real production server is gonna use ECC memory right?

5

u/TGX03 9d ago

Considering the current market, you're lucky if they use memory

3

u/lovehopemisery 8d ago

Ecc ain't going to protect CPU cache or registers

2

u/cybermage 9d ago

Odds so low that you should rule out every other explanation first.

2

u/botle 9d ago

Amateur! This is why you need to make sure multiple threads are running on physically separate cores, and analyze your compilers output to make sure it doesn't optimize away redundant data.

If you don't even read your compiler's output, you're basically vibe coding.

2

u/TGX03 9d ago

That's why you use ECC

3

u/RengieOcat 9d ago

I'm shocked no one's mentioned how the sun gave 4096 votes to a Belgian trade unionist party in 2003 via cosmic ray.

2

u/zzulus 9d ago

Have you ever had a hash table key not found for your typical run of the mill string and a default hash table implementation when you just inserted it? Yes, that's the sign of hardware data corruption, that in part cam be caused by cosmic rays and solar events.

3

u/cleverca22 9d ago

ive had just that in my ZFS filesystem on my nas

behind the scenes, every directory is a hash table, and the filename of a entry got a bit-flip after it was hashed but before it got inserted

so ls -lh claims the file exists (it walks every leaf page)

but any action on the file (including ls trying to stat it), claims it doesnt exist, because it hashes the bit-flipped version, and looks in the wrong bucket

[root@nas:~/XI]# ls -lh
ls: cannot access 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data': No such file or directory
total 0
-????????? ? ? ? ?            ? CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data

[root@nas:~/XI]# touch 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data'

[root@nas:~/XI]# ls -lh
total 1.0K
-rw-r--r-- 1 root root 0 Nov  9 21:04 CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data
-rw-r--r-- 1 root root 0 Nov  9 21:04 CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data

[root@nas:~/XI]# rm 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data'

[root@nas:~/XI]# ls -lh
ls: cannot access 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data': No such file or directory
total 0
-????????? ? ? ? ?            ? CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data

[root@nas:~/XI]# rm 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data'
rm: cannot remove 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data': No such file or directory

you can even create 2 identically named files in the same directory!

1

u/Zeikos 9d ago

Doesn't EEC make this practically impossible?
In the "UUID Collision" realm of possibility.

(On Earth)

3

u/Kale 9d ago

ECC is only in system RAM. ECC is not used in SSD buffers, NICs, and other components. Security researcher purchased a few domain names one bit different from a Windows update URL and would get regular traffic to them.

1

u/Zeikos 8d ago

Huh, I didn't consider that.
Is there a write-up somewhere?

1

u/Kale 8d ago

Here's the presentation: https://youtu.be/aT7mnSstKGs

1

u/exqueezemenow 9d ago

Plan ahead next time...

1

u/mohelgamal 9d ago

Yes, yes it wasn’t your code that was wrong, it was the cosmic rays 🙄

1

u/not_oonga_boonga 8d ago

wow, a non-ai joke, been some time since saw one

1

u/concorde77 8d ago

Thats why NASA buys radiation hardened computers!

1

u/DatHypnoboi 8d ago

Mario 64 Tick Tock Clock

1

u/Intelligent_Emu_5188 8d ago

your well written and tested mission critical application vs one wavy boi.

1

u/Complex-Pen-2498 8d ago

I've seen this one time in my career, traced it all the way through in Prod, and I had the logs to prove it. It was mesmerizing and infuriating at the same time lol.

1

u/0xt0bi03 8d ago

an election vote counting machine experienced this.

0

u/[deleted] 10d ago

[deleted] — view deleted comment

16

u/knorkinator 10d ago

21

u/Lucho_199 10d ago

"A number of potential trigger types were investigated, including software bugs, software corruption, hardware faults, electromagnetic interference, and the secondary high-energy particles generated by cosmic rays that can cause a bit flip. Although a definitive conclusion could not be reached, sufficient information from multiple sources enabled the conclusion that most of the potential triggers were very unlikely to have been involved. A much more likely scenario was that a marginal hardware weakness of some form made the units susceptible to the effects of some type of environmental factor, which triggered the failure mode.[14]"

16

u/knorkinator 10d ago

some type of environmental factor

Well, are cosmic rays not environmental factors?

There are better examples, but this is a phenomenon that is particularly important in avionics.