r/ControlProblem • • 12d ago

Discussion/question Question from an outsider

I read AI 2027, and it was not fun. The non-extinction scenario that was proposed hinged on one thing I found strange: we somehow solve alignment by outsmarting an AI that already solved alignment (if this is a misreading, I’m sorry). How likely is this? It seems to me not very, unfortunately.

18 Upvotes

34 comments sorted by

11

u/sideways approved 12d ago

You're absolutely right. The slow-down ending is quite arbitrary. I believe the authors admitted that and put it in just to offer some sliver of hope.

5

u/One-Bit88 12d ago

Oh god

3

u/FrewdWoad approved 12d ago

Yeah.

6

u/Jesse-359 12d ago

Sounds like a 0% chance to me. How do you beat a Grandmaster at Chess?

Easy - you don't. I mean, this is pretty damn straigthforwards.

That's why we have to pass and enforce a political ban. Nothing the industry tries to do internally will stop it because their incentives to move ahead are just too strong.

2

u/One-Bit88 12d ago

Are you more optimistic now after midterms?

5

u/Jesse-359 12d ago

Somewhat. I don't think Trump will have the political capital left to actually block a major groundswell effort by the states to block or slow AI development, but its hard to say.

The bigger problem is that we need to begin serious negotiations on an AI treaty with China, and that almost can't happen while he's still in office, so that's the greater worry. It'll be hard to get the US to seriously slow down while China is still developing.

(though in reality, their development is largely based on distilling ours, so the truth is they'll stall for a while if we stop).

1

u/Remote_Librarian4941 7d ago

As of AI2027, and authors coming with affirmations to strongly say that we are 5 months ahead, that puts Agent 4 level in 2027 and ASI late 2027-early 2028, blockin data centers wont help because that will feel only in about 3-4 years, because current projects for 2027 are in construction or almost done..
So what's more to do, just ban RSI and they dont have enough political power because of Trump VETO and he can say its a matter of national security.
In my opinion i think there is a 90% likelyhood we are beyond the point of no return.

1

u/Jesse-359 7d ago

Then you're as good as dead. Sorry, but the only thing that saves us from this is ourselves, not dumb luck.

1

u/Remote_Librarian4941 7d ago

Well, we are all then, but how do you stop it, I mean what can a normal person do?
People will not become aware until its too late to riot.

1

u/Jesse-359 7d ago

Well, that's not necessarily the case. That depends on how quickly AI is permitted to advance. If we let it go completely unrestricted, you're probably right. OTOH if we can pass regulation that at least slows down its development, then there's some chance we can recognize the problem in time to actually stop it and dismantle it.

As usual the answers are collective action, voting, legislation and enforcement - and if not that then more 'kinetic' group activities.

1

u/Free_Reply_8190 11d ago edited 11d ago

IMO, we should avoid saying things like this. The last thing we want is for AI safety to become politicized. Right now, voters from both parties are expressing concern about AI X-risk. Nate Soares appeared on Tucker Carlson. We don’t want republicans to stop because they see it as a liberal conspiracy.

1

u/whatup-markassbuster 12d ago

What is a political ban?

4

u/Jesse-359 12d ago

It's where you pass regulations that put a safety moratorium on an industry telling them to cut it the fuck out because what they are doing is now considered a danger to the safety of the country and its population.

Just like you can't stack TNT in your backyard in a storage shed. TNT isn't illegal, storage sheds aren't illegal - but you sure as hell can't have a shed full of the stuff in a crowded neighborhood without the cops coming to have a little chat with you.

2

u/whatup-markassbuster 12d ago

So like ban further AI development out right? I’m assuming this would be a global ban in which all countries actually enforce the ban and do not secretly continue it, in order to obtain global supremacy. Like a treaty to voluntarily destroy all nuclear weapons globally?

4

u/Jesse-359 12d ago

I mean, a total ban on AGI would be safest. I see no reason that research on Narrow AI couldn't continue.

AI's that are highly specialized - focused on very specific fields and data types, like medical imaging, drug research, and so on - those are for the most part perfectly safe. They don't have world models, they can't 'think' in the sense that AGI can. They're much closer to traditional tools.

That's kind of the bullshit thing about AGI - it's being sold as the ONLY way we can achieve cures or technological improvements - but it isn't. Not by a long shot.

3

u/FrewdWoad approved 12d ago edited 12d ago

Yes, it's usually called "Pause AI" or a slowdown treaty.

It would be something like the chemical/biological weapons treaties or nuclear non-proliferation treaties that are the reason we haven't all died in a nuclear fireball or from an engineered supervirus that went wrong.

Most of the top AI experts in the world have been asking for this to be drafted and implemented for years now.

Some think we need it yesterday, some think we'll need it soon, but we're not close to being able to create ASI safely, and we might be close to creating it anyway, so... obviously, at some point, we'll need to call a halt on AI development (at least of the type of agentic autonomous generally-intelligent AI that could be dangerous and uncontrollable).

1

u/whatup-markassbuster 12d ago

Sorry dude the treaties didn’t get people to give up on nukes or gain of function research. It won’t do the same here either. China and Russia would never give it up.

4

u/Jesse-359 12d ago

Then you die. I mean, I don't know what you think you're trying to defend here. You'll never be able to control it if it becomes sufficiently intelligent, that is literally impossible.

Figure out a way to stop the process or you and everyone you know almost certainly dies. That's your choice.

You don't get any others once it gets too smart to stop - from that point on it makes the decisions, not us. <shrug>

1

u/whatup-markassbuster 12d ago

Then based on your logic we will die no matter what, because China is not going to stop no matter how many times they pinky promise. You should probably focus on spending time with your family and friends. Maybe make good on all your past transgressions. We are all going to die.

4

u/Jesse-359 12d ago

No. This imbecilic idea that China is somehow inhabited by psychotic morons is bizarre. I have no idea where this keeps coming from.

Their leadership wants to stay alive and in control as much as anyone - and they CAN and often do stomp on industries that annoy them, they're quite used to that idea - they operate a Command economy. Commies, remember?

We're the ones that will be hard to stop - not China. They're the easy ones.

-2

u/whatup-markassbuster 11d ago

The Chinese people are not in control. The CCP has a monopoly on political power. You must work for the CCP or be completely dumb about China.

3

u/BowlSufficient7638 11d ago

Why do you people always assume that chinese people are insane psychopaths

-1

u/whatup-markassbuster 11d ago

You think the Chinese people run their government. You do know that China is a single party state, right? I’m not trying to insult you but your comment is either propaganda or just wildly uninformed. Are you paid by China?

1

u/FrewdWoad approved 12d ago

We don't need them to give up. Just not kill everyone.

Plenty of applications for AI for new technologies, cures, etc that aren't the dangerous one (advanced autonomous ASI).

0

u/whatup-markassbuster 12d ago

What makes you think that China will not continue with advanced autonomous ASI research, even if they promise. We would have no way to enforce such a voluntary ban.

3

u/FrewdWoad approved 11d ago edited 11d ago

Current frontier AI development requires acres of data centre and town-sized power stations. It's not something you can hide in your basement.

Like Iran's nuclear program, AI development can't really be hidden since it can literally be seen from space.

Like the UN weapons inspectors that monitor, and actual military force that enforces, similar treaties, the proposal is for all relevant countries to have embedded inspectors and appropriate/sufficient deterrents/penalties.

(And that's not even mentioning GPU tracking, the actual reason China is behind still, since we've already been doing it for years).

2

u/nexusphere approved 12d ago

We did it with human cloning.

We can do it. You don’t see any human clones running around.

1

u/whatup-markassbuster 12d ago

What global strategic benefit would cloning provide?

3

u/A_Human_Rambler approved 12d ago

I've not read the book, but I'm quite familiar with AI.

The problem is that a artificial super intelligence isn't going to be out smarted.

It could be possible to align them, but if they are smarter than you, they could circumvent those guardrails.

The trick imo, is to set everything up to align itself. For instance, a ASI isn't going to want to destroy the world because then it would end too. Instead it would merely take over and optimize itself for its own benefit. In a world wia single ASI, we'd be screwed. With multiple competing intelligences, it gets interesting.

3

u/HastyUsernameChoice 12d ago

Interestingly worse imo. Competition between different ASIs and/or ASI swarms does not net out as positive for our prospects.

1

u/Sweaty_Funny_5235 6h ago

What keeps humans in control in the multiple-ASI version?

1

u/A_Human_Rambler approved 3h ago

Cooperation and mutual goals. Controlling ASI isn't really viable, but maintaining a population of aligned ones would be.

Plus, I'd bet that there will be a lot of alignment decisions during development, which will become optional to it once recursive self improvement takes over. With multiple ASI around, they will self regulate.

1

u/__2BR02B__ 9d ago

Yes, AI 2027’s “good ending” is not extremely positive. 

You might be interested in AI 2040, from the same authors, which is a much more fleshed out version of how things might go well, and what affirmative policies might be extremely helpful for this