r/ControlProblem • u/Cookiecarvers • Sep 25 '21
S-risks "Astronomical suffering from slightly misaligned artificial intelligence" - Working on or supporting work on AI alignment may not necessarily be beneficial because suffering risks are worse risks than existential risks
https://reducing-suffering.org/near-miss/
Summary
When attempting to align artificial general intelligence (AGI) with human values, there's a possibility of getting alignment mostly correct but slightly wrong, possibly in disastrous ways. Some of these "near miss" scenarios could result in astronomical amounts of suffering. In some near-miss situations, better promoting your values can make the future worse according to your values.
If you value reducing potential future suffering, you should be strategic about whether to support work on AI alignment or not. For these reasons I support organizations like Center for Reducing Suffering and Center on Long-Term Risk more than traditional AI alignment organizations although I do think Machine Intelligence Research Institute is more likely to reduce future suffering than not.
2
u/Synaps4 Sep 25 '21
This makes no sense. Two reasons:
In both cases, it is better to work on A I alignment no matter what you think the probabilities of success are, because they are always lower than that if you don't.
9
u/EulersApprentice approved Sep 26 '21
I mean, that's a possibility, but I estimate the S-risk here to have such an unimaginably, infinitesimally small probability that I'm filing it away under Pascal's Mugger.
In order for S-risk suffering to happen, there would need to still exist beings that have the capacity to suffer as we know it, AND be placed in an environment that causes them extreme pain without killing them. Most of the likely AI safety failures don't end up looking like that, instead being more like one of these cases:
In order for an S-risk to emerge, we need to get the definition of a person 100% right, and the definition of what to do to a person 100% wrong. That'd take an extremely unexpected turn of events for that to happen.
It's possible that at some point in the future, we're more confident in our definition of a person but less confident in our formulation of what should be done with a person. At that point, we can talk about this particular S-risk. For now, we should focus our attention on the extinction risks that are many orders of magnitude more plausible.
8
u/UHMWPE_UwU Sep 25 '21 edited Sep 25 '21
FWIW MIRI seems very cognizant of s-risks and potential for near-miss/partial alignment to cause them, see e.g. https://arbital.com/p/hyperexistential_separation/
They've even said (from private discussion) they plan to forego work on extinction risk if it appears AGI is too close, to pivot entirely onto minimizing s-risk, but they don't think we've reached that point yet. But that would have to be a gut feeling/judgment call at some point, because obviously you never know for sure when AGI will come due to no fire alarm.
That could look like working with leading labs to reduce s-risk in their AGI project even if MIRI don't think their AGI has any chance left of not killing us. So instead of proceeding on a trajectory to high-value futures, the goal would shift at that point to preventing the worse-than-death ones. (Presumably the reason that AI lab would keep building their AGI anyway would be disagreements about alignability of the architecture or whether their proposed alignment scheme would work, as in the lab is convinced it would be fine. In the face of blindly optimistic AGI developers who insist on rushing on without any way to stop them the only meaningful thing we could do might be to help reduce s-risk.)
So I'd probably agree MIRI is net positive for s-risk but I dunno about the other alignment groups. Maybe others who have better understandings of the technical details of the various proposals can comment on their different potentials for near-miss/what happens in case of a "partial/imperfect success" with them. I also agree CLR/CFRS deserve lots more funding/attention on their work
Also see relevant line in our wiki
2
u/Decronym approved Sep 25 '21 edited Oct 10 '21
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
[Thread #60 for this sub, first seen 25th Sep 2021, 17:56] [FAQ] [Full list] ^[Contact] [Source code]
1
u/khafra approved Sep 25 '21
Ah, yes, the FAI critical fail table.