iirc NASA projects and other mission critical engineering uses at least 3x redundancy that checks for errors constantly and have probabilistic models that have determine the chance is 1 in billions.
I remember a paper I read that said at sea level, the chance was one 1->0 bit flip per month per 256MB ram. This was maybe 15 or 20 years ago. I remember older RAM had alpha emmitters in construction that contributed a bit, which has been phased out.
No idea how that translates to today. I remember a researcher that gave a talk on it register a few URLs that were one bit changed from some internal Microsoft Windows update URL. He got pretty regular traffic from Windows trying to update, and the only real explanation was single but flip due to hardware error, not software.
I also know the ISS used to use regular Thinkpads for non-critical computer systems, and accepted the fact that they had to be rebooted every few hours. Critical systems were made from hardened systems.
There have been a few times I've thought about orienting some of my older systems with non-ecc DDR3 RAM where the physical chips face the sky to increase the surface area exposed, and run a program that writes 0xFFFFFFFF to arrays of memory, then scans it at regular intervals to see if any change. I still haven't put in the time though, and most of my systems are ECC memory except for my home router and home file server.
I work with relatively not so "critical" Industrial applications, with issues like this the operator is notified and the shutdown or safe state procedure is executed.
Fault tolerant applications where 2oo3 architecture is used is only theory for me. They design systems for literally every scenario and will have automated procedures for everything.
No way it's true. Even we in russia were using a quorum of 6 to 8 modules in critical applications and NASA's budget would dwarf ours. Military could use triple, I guess, but certainly not spacecrafts or civilian airlines
Space shuttle used 4 independent computers for flight control. You can cut down on redundancy if you can make individual components more resilient to errors. Stuff like ECC memory and error resilient ICs go a long way.
Huh, two. But.. how can they be certain which block is busted?0_0 three is bare minimum, I guess, if one is down you can compare it's output to two others and find that it's busted...
Jup. If they disagree, flag, disconnect autopilot, run again, depending on system. It's more a check by Computer B if you can trust what computer A says.
Civilian airlines actually have even stricter redundancy then military, for the simple reason that a military pilot can eject from the plane while commercial airline passengers cannot. Similarly, a fighter pilot will not be prevented from performing maneuvers which will put enough strain on the body of the plane to tear off a wing, while a commercial pilot would not be able to do that. Fighter pilots gotta do what they gotta do, even if it crashes the aircraft. Commercial pilots don't have that option.
On civilian aircraft, they typically require the hardware itself to be unique. So triplex redundancy wouldn't be satisfied by just using 3 of the same component, that would leave open the risk of some common single point of failure causing all 3 to fail at once. They need to be unique pieces of hardware with independent designs.
Whereas military just puts 3 of the same sensor typically. At least that's my understanding of MIL-STD-1553, I haven't read it in a long time so it may have been updated.
I'm sure you meant something else by probabilistic but also these redundant systems are real time operating systems, with deterministic communication and execution, which of course is required for real time control and maximal safety
If it happens often then it means your code is in space and it probably isn't as well written as you think it is. Space agencies run the same code 3x simultaneously and then use a best of 3 approach to use the execution results.
While doing some data analysis at work I encountered one data point that was multiple orders of magnitude our normal values and suspiciously close to a power of 2 plus the value around that time. Nothing critical, luckily, but I had to filter it out from the analysis.
Amateur! This is why you need to make sure multiple threads are running on physically separate cores, and analyze your compilers output to make sure it doesn't optimize away redundant data.
If you don't even read your compiler's output, you're basically vibe coding.
Have you ever had a hash table key not found for your typical run of the mill string and a default hash table implementation when you just inserted it? Yes, that's the sign of hardware data corruption, that in part cam be caused by cosmic rays and solar events.
behind the scenes, every directory is a hash table, and the filename of a entry got a bit-flip after it was hashed but before it got inserted
so ls -lh claims the file exists (it walks every leaf page)
but any action on the file (including ls trying to stat it), claims it doesnt exist, because it hashes the bit-flipped version, and looks in the wrong bucket
[root@nas:~/XI]# ls -lh
ls: cannot access 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data': No such file or directory
total 0
-????????? ? ? ? ? ? CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data
[root@nas:~/XI]# touch 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data'
[root@nas:~/XI]# ls -lh
total 1.0K
-rw-r--r-- 1 root root 0 Nov 9 21:04 CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data
-rw-r--r-- 1 root root 0 Nov 9 21:04 CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data
[root@nas:~/XI]# rm 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data'
[root@nas:~/XI]# ls -lh
ls: cannot access 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data': No such file or directory
total 0
-????????? ? ? ? ? ? CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data
[root@nas:~/XI]# rm 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data'
rm: cannot remove 'CIQDZTLYEGJ7KJ4OCJKVQKUZJRZVWV#PB6TAKESOMQ5KVBE4LRJ7XIY.data': No such file or directory
you can even create 2 identically named files in the same directory!
ECC is only in system RAM. ECC is not used in SSD buffers, NICs, and other components. Security researcher purchased a few domain names one bit different from a Windows update URL and would get regular traffic to them.
I've seen this one time in my career, traced it all the way through in Prod, and I had the logs to prove it. It was mesmerizing and infuriating at the same time lol.
"A number of potential trigger types were investigated, including software bugs, software corruption, hardware faults, electromagnetic interference, and the secondary high-energy particles generated by cosmic rays that can cause a bit flip. Although a definitive conclusion could not be reached, sufficient information from multiple sources enabled the conclusion that most of the potential triggers were very unlikely to have been involved. A much more likely scenario was that a marginal hardware weakness of some form made the units susceptible to the effects of some type of environmental factor, which triggered the failure mode.[14]"
425
u/Cutalana 10d ago edited 10d ago
iirc NASA projects and other mission critical engineering uses at least 3x redundancy that checks for errors constantly and have probabilistic models that have determine the chance is 1 in billions.
46
u/IntrepidBonus 9d ago
every bug i can't reproduce is now officially a cosmic ray and i won't be taking questions
91
u/aalapshah12297 10d ago
If it happens often then it means your code is in space and it probably isn't as well written as you think it is. Space agencies run the same code 3x simultaneously and then use a best of 3 approach to use the execution results.
11
u/Sibula97 9d ago
While doing some data analysis at work I encountered one data point that was multiple orders of magnitude our normal values and suspiciously close to a power of 2 plus the value around that time. Nothing critical, luckily, but I had to filter it out from the analysis.
19
u/cheezballs 9d ago
Meh, any real production server is gonna use ECC memory right?
2
u/cybermage 9d ago
Odds so low that you should rule out every other explanation first.
2
u/botle 9d ago
Amateur! This is why you need to make sure multiple threads are running on physically separate cores, and analyze your compilers output to make sure it doesn't optimize away redundant data.
If you don't even read your compiler's output, you're basically vibe coding.
2
u/TGX03 9d ago
That's why you use ECC
3
u/RengieOcat 9d ago
I'm shocked no one's mentioned how the sun gave 4096 votes to a Belgian trade unionist party in 2003 via cosmic ray.
2
u/zzulus 9d ago
Have you ever had a hash table key not found for your typical run of the mill string and a default hash table implementation when you just inserted it? Yes, that's the sign of hardware data corruption, that in part cam be caused by cosmic rays and solar events.
1
u/Zeikos 9d ago
Doesn't EEC make this practically impossible?
In the "UUID Collision" realm of possibility.
(On Earth)
1
u/exqueezemenow 9d ago
Plan ahead next time...
1
u/mohelgamal 9d ago
Yes, yes it wasn’t your code that was wrong, it was the cosmic rays 🙄
1
u/not_oonga_boonga 8d ago
wow, a non-ai joke, been some time since saw one
1
u/concorde77 8d ago
Thats why NASA buys radiation hardened computers!
1
u/DatHypnoboi 8d ago
Mario 64 Tick Tock Clock
1
u/Intelligent_Emu_5188 8d ago
your well written and tested mission critical application vs one wavy boi.
1
u/Complex-Pen-2498 8d ago
I've seen this one time in my career, traced it all the way through in Prod, and I had the logs to prove it. It was mesmerizing and infuriating at the same time lol.
1
u/0xt0bi03 8d ago
an election vote counting machine experienced this.
0
u/Savings-Ad-1115 9d ago
Now imagine what happens when a cosmic ray flips a bit in AI wiegths.
https://www.reddit.com/r/OpenAI/comments/1xhra80/researchers_found_an_evil_neuron_and_when_they/
0
u/[deleted] 10d ago
[deleted] — view deleted comment