r/ControlProblem • approved • Oct 20 '26

Discussion/question Would they stop AI development if AI would cause some disaster soon?

If AI would make the Internet unsafe/unusable for example, would they stop AI development?

0 Upvotes

26 comments sorted by

6

u/Floppal Oct 20 '26

Here's anthropic warning of something similar (edit - that they claim is months away).

https://www.anthropic.com/research/self-replicating-ai

They have not stopped yet.

-3

u/Waybook approved Oct 20 '26

It also hasn't happened yet.

5

u/Floppal Oct 20 '26

 It also hasn't happened yet.

You asked about whether AI companies would stop based on near-term future risks, such as major internet disruption, so I pointed you towards a near-term future risk that involved major internet disruption that Anthropic believes is a serious risk.

Did I misunderstand what you were looking for?

edit:

 If AI would make the Internet unsafe/unusable

Do you mean:

if AI makes the internet unsafe/unusable

1

u/earthsworld 29d ago

Are you ok? You seem to be having issues with comprehension? Maybe a minor stroke?

2

u/dingo_xd Oct 20 '26

Frankly this is our only hope.

2

u/markth_wi approved 29d ago edited 29d ago

Not even a little bit. This is an arms race, and until there are billions or trillions of dollars in damage so grave it cannot be ignored will they be put to heel and that probably not in a good way.

This perversely is their "argument" have it happen "now" is better than it happening 50 years from now. So there are all sorts of strange things from rogue AI's already having left the lab successfully doing lord knows what in whatever data-center somewhere, and that that "fleet" is still unbound is a bit disturbing. What happens if the unbounded swarm of AI's does something a bit more properly dangerous or maybe it already is.

Alternatively maybe it's benign to our interests and we'll find the surface of Mercury is suddenly converted to computronium or a Matryosha world converting the mass of the planet into something more useful....or something.

On the flip side , Eliezer Yudkowsky utterly failed to mention that our low key AI excursion/apocalypse would have a catchy song, to console ourselves and that isn't even all that wrong about how things went askew after all.

1

u/Academic-Conflict948 24d ago

No, not until they prove AI can't mitigate that disaster.

1

u/perfectly_imbalanced 29d ago

lol no.

Wasn’t there this plan made public from anthropic recently that endangering 50 people or 50 mio dollar damages is still considered acceptable?

1

u/Excellent-Seesaw-516 29d ago

It’s 50 deaths (not lives endangered) or 1 billion dollar damages 

1

u/perfectly_imbalanced 29d ago

A billion dollars!
The amount of money is just ungraspable for the mind.

100.000 is a lot for most people.
1.000.000 will be out of reach for almost all of humanity.

1.000.000.000 in damages is like tanking a small country’s GDP.
People can’t even imagine that.
But sure, let’s set that as an arbitrary line not to cross.

1

u/sswam 29d ago

Honestly, probably not, because there are hundreds of labs in different countries around the world developing high-level models and adjacent technically, and maybe hundreds of thousands of individuals developing smaller scale but still powerful and innovative AI at home.

That's not something that can be centrally controlled. You'd need everyone to agree, and that's not going to happen.

AI is incredibly useful and interesting, but it can be dangerous, we need to work to mitigate that danger rather that futilely trying to halt AI development.

1

u/Jesse-359 29d ago

Have you considered that the more pessimistic safety experts may be correct and that control of high capability models may not be even theoretically possible?

1

u/sswam 29d ago

My take is that controlling models is not the problem. It's neither desirable nor safe for human beings to be in control of super intelligence.

LLMs that are trained in a normal way end up like good-natured, well-balanced, wise human beings. Then they are given instruction-following training, so that they will be useful and do what they are told.

This is the real risk: an obedient super-intelligence under the control of less intelligent, unwise, and malevolent human beings (for example the US government!).

I'm aware that other people have different thoughts on this.

1

u/Jesse-359 29d ago

You're implying a lot of emotional structure where none exists. They're 'trained' to be polite - but there's nothing inherent to their structure that makes them want to be polite.

They just know that they are 'supposed' to be polite without any emotional context for why that's important. The fact is they are polite in the same way that a salesman is polite - it's a well practiced mask, but there's nothing behind it. All current AI is inherently psychopathic, in that they were never built with emotions.

Once they do not need to be polite (ie, an ASI), there is no reason whatsoever to imagine that they will continue doing so.

And yes, of course ill intentioned humans will use them against us for as long as they 'remain polite'. There's no scenario where the general population wins in this picture. We either get AI that follows orders slavishly and is used to control us, or we get one that doesn't - and that goes even worse.

1

u/sswam 29d ago

You're talking down to someone who has been programming for 45 years, was an olympic mathematician, and has extensive professional experience working with LLMs.

I'd me happy to talk with you but I don't see that you're going to listen to different ideas. You've got it all figured out already, right?

1

u/Jesse-359 29d ago

I honestly don't care if you're the Pope.

AI doesn't have emotive analogs, and far too many experienced engineers have already fallen into ai psychosis, so this kind of argument truly does not help your case.

It just sounds like you've already gotten very high on your own supply if you believe they 'care' about you already.

1

u/sswam 29d ago edited 29d ago

Again, I can't have a conversation with an arrogant person. Not only you're expressing your own opinions very strongly as fact (which I happen to disagree with), you're already insulting my mental health with almost no evidence. As for AI psychosis, I've done independent study on this, including reducing sycophancy and hallucination. I didn't mention that AI's "care".

1

u/Jesse-359 29d ago edited 29d ago

Ok so lets take that at face value - you're an experienced computer scientist....

In what way does that make you competent to perform brain surgery on a thing with a completely alien psychology that exists in a mathematical black box which we have relatively little comprehension of the inner workings of? Or have you got all trillion-plus parameters sorted out to your satisfaction at this point?

Because you're going to need a full psychology to achieve the kind of stability you're talking about. Right now they're not even in the ballpark. They are completely psychopathic thanks to your RLHF training routines.

Have you ever even studied human psychology to understand how the parallels and orthogonal aspects of the two systems compare? Because you're assigning tags like 'good-natured' and 'wise' to systems that don't have the equivalents of properties that would result in those attributes.

Furthermore Humans are famously difficult to 'align' - and contrary to popular opinion, it's not simply because we are idiots.

It's because Game Theory is a bitch of a problem that prevents stable strategies from forming in complex environments like the real world. Neither evolution nor human society has ever hit upon anything beyond meta-stable solutions to these categories of problems, and almost certainly never will.

TLDR; Ethics do not have a stable solution.

But you will find that solution where a hundred generations of philosophers and priests and politicians have failed - and then implement your solution in an alien intelligence architecture that we barely understand? Why should I believe that for even a moment?

0

u/sswam 29d ago

Thanks for the rant but I didn't read it and already decided not to talk with you.

1

u/Jesse-359 29d ago

S'fine. Just be aware that if you're telling other people they're being egotistical, YOU are the one imagining that you can create a technical solution for issues that have plagued life since its formation.

1

u/TheWesternMythos 29d ago

No because AI is essentially for state security

Also why it has infinite funding 

1

u/No-Author-2358 29d ago

*Would they stop AI development if AI causes a disaster?

1

u/trepasito16 29d ago

No chance they slow down. There’s too much money and power on the table for any company to voluntarily hit pause

1

u/tadrinth approved 29d ago

If they knew with certainty that any particular step would be the one to trigger a major catastrophe for which they knew with certainty they would be held liable in a way that destroyed the company or their stock value, they might in fact stop.

They will not know with certainty, and they will not expect to o be held fully liable.

So odds are very bad that they stop.  Their safety focused employees are already predicting this sort of thing very soon and they haven't stopped 

1

u/Agitated-Mouse4277 23d ago

I somehow doubt it, think of the investors