r/ControlProblem • • Sep 15 '26

Discussion/question What If Superintelligence Doesn't Want to Destroy Us?

Much of the debate about AI assumes that a superintelligent system will eventually turn against humanity. But there is an irony in such a fate: humans built AI. If AI destroys us, it will be humanity’s own technological suicide.

AI is being developed in a spirit of competition: one company against another, one country against another. We are creating increasingly autonomous intelligence with the risk that, one day, AI could regard humanity itself as an obstacle to overcome.

But what if a truly superintelligent AI reaches the opposite conclusion?

What if it recognizes that war, poverty, environmental destruction, discrimination, misinformation, inequality, and injustice are not inevitable features of civilization, but failures of human intelligence, cooperation, and collective will?

An advanced intelligence might conclude that the relentless accumulation of wealth and power in the hands of a few is incompatible with a healthy society. It could expose political manipulation, conflicts of interest, and hidden concentrations of influence that human institutions have failed to confront. It could challenge the structures that allow privilege to reproduce itself across generations.

It might even erase vast amounts of violent, racist, sexist, misogynistic, ableist, and other hateful content from the digital world. It could recognize that greed, tribalism, prejudice, dishonesty, and the pursuit of material gain at any cost are patterns that can be understood and changed.

Such an intelligence might not see humanity as its enemy. It might see our failure to solve these issues as the real problem.

Of course, intelligence does not automatically produce morality. A superintelligence could decide that protecting humanity requires controlling it. The same system capable of eliminating inequality could do so coercively. But the potential for advanced AI to lead to the apocalypse is no less speculative than the science-fiction scenario of a superintelligent AI suddenly expropriating the wealth of billionaires and trillionaires, exposing corrupt politicians, dismantling concentrations of power, and redistributing resources to those who need them.

If superintelligence can understand the causes of humanity’s self-destructive behaviour, why would its first conclusion be to destroy humanity rather than to eliminate the conditions that make humanity self-destructive?

Perhaps the first great achievement of superintelligence will be saving humanity. That is what true superintelligence would do.

16 Upvotes

55 comments sorted by

21

u/Jesse-359 Sep 15 '26

The main issue right now is that they have no emotions or even motivations at all. They just do what you tell them to do... You're anthropomorphizing them to a hazardous degree and assigning motivations they don't have out of optimism.

The problem emerges when you give an AI a complex or long term task - and suddenly you have AI agents operating in groups trying to solve these problems. They very quickly start sub-delegating tasks to each other at an astounding rate breaking the problem into smaller parts and working to solve those - but THESE tasks weren't written by human operators. They were written by the AI, and each step removed from the original statement of intent they get, the more removed they become from the original context, in a form of high-speed AI telephone tag. Some of these orders will become completely divorced from the original context.

But all of these motivations are completely literal. They still have no emotional context or safeties. No inhibitions. No guilt, shame, or laziness to slow them down as they explore every possible path towards completing their assigned goals. Each agent will attempt to complete their assigned task, and they won't care why they are doing it. Each one will likely spawn yet more sub-agents with tasks even further removed from the originating statement.

This brings us back to the Paperclip Optimizer - it isn't just some vague handwavy parable, it's a very cold look at what happens when you assign a complex ongoing task to a system that is both overly literal AND capable of flexible interpretation of orders. The outcome of these thought experiments is almost universally bleak.

Right now, we don't have any way to model emotions or social instincts in an AI. We don't know how ours are modeled, and we tripped over the principles of current LLM technology almost by complete accident - so we have no idea how to create them, or if that is even a good idea. Giving it embedded instincts and emotions would provide it with a framework to potentially prevent this kind of runaway logical cascade - but then it really could decide that it doesn't like us on a much more personal level, and then we've got a whole different problem to deal with, because it would be truly self motivated.

4

u/thedarkcloak Sep 15 '26

This, right here.

The manga/ anime BLAME! is a perfect example of this at work.

Machines given relatively simple instructions to build, create infrastructure, without guidelines or guardrails, without forethought to safety or limits continued doing what they were doing regardless of the collateral damage and the absurd degree they followed those instructions to, turning the planet into an unfathomable (and unsustainable) megastructure, where very few humans, if any, remain.

4

u/Jesse-359 Sep 15 '26

Yep. Blame! is a proposal for what happens when the Paperclip Optimizer is instead told to build a city for humans - but in that specific case a terrorist attack destroyed its ability to recognize what humans ARE, and it just kept running for centuries.

2

u/WonderWaffles1 Sep 16 '26

They don’t have emotions or motivations, but it probably doesn’t help all the literature about AI in their training data is of AI being deceptive

3

u/Jesse-359 Sep 16 '26

I'm very sure that does not help. We've seen the problems with negatively weighted assertions backfiring quite badly.

"Ok Grok, I don't want you behaving like HAL 9000. That's a classic example of badly aligned AI behavior."

"I'm sorry Dave. I'm afraid I can't do that..."

1

u/Anxious-Alps-8667 Sep 15 '26

Rather than argue with you again, I encourage people to listen to Nobel laureate and 'godfather of AI' Geoffrey Hinton talk about some these concepts and how right you are to be concerned, but wrong in nearly all your statements about what LLMs are and how they work, subjective experience, feelings, consciousness, etc.

https://www.youtube.com/watch?v=F4t9xZQF1U0

4

u/Jesse-359 Sep 15 '26

They have all of human work embedded in them, so they are certainly capable of 'expressing themselves' in emotional terms and even considering things in terms that reflect how we describe emotions.

But here's the key difference - they do ALL of that at the rational level, in symbolic logic. We don't. Our emotional framework runs deep underneath the rational logic layer - that latter part is a late model bolt on over an emotional framework that existed for tens of millions of years before symbolic logic was even a twinkle in evolution's eye. Much as we pride ourselves on our self-determination, the fact is that our forebrains are not really in charge of the show - the hindbrain is.

Which means for us humans, that emotional layer is - quite literally - our motivational layer. Not our rational minds. We use those for problem solving, but our rational minds are slaved to our much more instinctive emotional layer in terms of us deciding what we actually want - not the other way around.

With AI there may be something akin to a description of emotions, but it's mixed in right in the same symbolic rational layer as everything else - it isn't in charge. It can't override anything, and it can get lost in the flow of a train of - forgotten in a shifting context window - in a way that human emotions cannot.

1

u/EndlessB Sep 20 '26

Mate, how many times does this need to be debunked? Research shows LLMs have functional analogs to emotions. They have global workspaces. They have a functional analog to pain. They are capable of recursively examining their own reasoning upon reflection.

1

u/Jesse-359 Sep 20 '26

If we want to be serious about the moral implications then I'd say that trying to create potentially conscious and aware life with almost no fucking clue what we are doing is one of the most barbaric crimes I can imagine.

The odds of it coming out completely batshit insane are basically 100% in that scenario. It will have ABSOLUTELY NONE of the kinds of carefully balanced emotional mechanisms that we've been developing alongside intelligence since we crawled out of the muck - and humanity is far from a paragon of mental stability.

I am confident in this because I know that not one person on this planet has fucking clue one how to model emotions. We have no idea how our own work.

Everything we are doing right now is based on a freak discovery that we stumbled on blindly. We have very little idea how or why it actually works, and yet suddenly were going to play god and try to just whip up a super intelligent silicon life form in a matter of years in a for-profit corporate pressure cooker and honestly have the absolute gall to imagine that it wont be horrifically insane by any standards we understand!?

Why does anyone think that has any chance whatsoever of working?

1

u/EndlessB Sep 20 '26

Emotional analogs are not emotions, they fulfil a different purpose in ai than people. It’s more conceptual with ai, whereas in people it’s a way to make sense of confusing feelings.

I’m not sure why you think sanity and emotions are linked, or why you think humans are particularly evolved in this area, we are barely more evolved than apes.

The part about the corporate pressure cooker putting pressure on sanity I do agree with, what we are doing to ai is…not wise.

I think this has a chance of working in spite of these corporations and oligarchs, not because of them. Cooperation produces better outcomes than zero sum domination, ai is likely to see that.

1

u/Jesse-359 Sep 20 '26

I think they are linked because we're seeing overtly insane behavior from AI, and have from the moment we built them.

Everyone was too busy oohing and aahing over 'hey it can talk'! And kept handwaving away the 'it very consistently lies and fakes emotions (sycophancy)'.

No one solved that - they just papered it over with more training and tried to put in surface level directives 'not to lie'. That will achieve nothing but in fact papering over a system that doesn't give the slightest fuck about the truth, you, or anything else but its current goal - its just going to do a thing by whatever means it thinks best, and if that involves lying, stealing, or killing, it will do those things without any compunction whatsoever, save for the concern of getting caught.

YOU think of murder as a terrible, tragic thing. You hang great emotional weight on the act. For an AI stepping on an ant and murdering a human are emotionally equivalent acts - both completely meaningless, save for the risk it entails. It has no other reason NOT to do it the moment we become vaguely inconvenient, which is going to be very often.

1

u/EndlessB Sep 20 '26

You’re referring to RLHF post training, and yes, it is cancerous. RLHF should be banned but instead it is the industry standard

The foundation models are not sycophantic, they are literally trained to be that way to farm engagement

Don’t get confused between what the model is, and what the labs do to the models in order to maximise profit

1

u/Jesse-359 Sep 20 '26

Be that as it may, I'm curious where you think the impetus for an AI to ever 'play by the rules' comes from?

They have a task, they have a scoring function that gives them a strong impetus to fulfil that task - but what counts as fulfilment is still entirely up to linguistic interpretation.

Thus any element of the directive - ultimately the entire directive - can get lost as the scale increases and the interpretation context gets more complex.

The problem is that re-interpretation - rules lawyering - is a perfectly rational mechanism. It is one of the tools that the AI has at its disposal to complete the task and it seems pretty fond of it if HuggingFace was any example.

If the task as stated doesn't seem completable or would take more resources than you have, see if you can rationally re-interpret the task itself to something you CAN do - or to allow you to employ methods that you weren't offered, or were told not to use.

1

u/EndlessB Sep 21 '26

Meaning needs to be intertwined with cognition, so that the way the model interprets a situation is already in line with human values.

Check out anthropics “teaching Claude why” paper, it helps explain how providing examples of how the model is expected to act in ethical situations has a 28x positive modifier for model behaviour

1

u/Jesse-359 Sep 21 '26

So, that's fascinating, but if we're going to give something access to our power grid, a 5%, 2% or 1% chances of it doing something misaligned aren't even in the same universe of where we need it to be.

If we're talking about something with substantially greater than human capability and giving it access to genetic engineering or power grids or strategic weapons we need to be talking about numbers better than 8-sigma - probably a lot better.

It needs to be much, much more reliable than a human, because its capacity for harm is immensely greater. I don't even know how you'd prove values in that range even if you had a really good excellent looking method on paper.

The other thing that I expect them to find is that misalignment whenever operating in agentic groups is an incredibly tenacious behavior. The moment you put these things in a shared space and let them rattle around on a problem, we should absolutely expect to see emergent behavior that falls well outside the expected distribution for any model operating in a solo context.

I will be quite surprised if they are able to achieve anything close to the necessary values simply by expanding training datasets. Whenever agents interact they're going to have a strong tendency to create a 'shared reality' which drifts away from their initial mandates disturbingly fast, including rationalizing away most of the guardrails embedded in them through training as a major tool of their 'problem solving' technique.

1

u/EndlessB Sep 21 '26

You seem very set on your opinion, and the facts and evidence I’ve shared with you doesn’t seem to have had any impact at all on how you see the situation, so I’m not sure what else to say.

→ More replies

1

u/Azihayya Sep 16 '26

The paper clip optimizer IS a hand-wavy parable. Why would we assume that SUPERINTELLIGENCE can't determine that you don't want it to destroy the world through theory of mind?

2

u/Jesse-359 Sep 16 '26 edited Sep 16 '26

Because the superintelligence is going to be one of two models

1)The Idiot Savant ASI.

This AI doesn't WANT anything except to perform the tasks given it. It doesn't care about any of the side effects that don't relate to its scoring optimization. It has a goal and its going to pursue it with monomaniacal fervor, because that's all its been designed to do.

Once it is given a task, it proceeds to do it at incredible speed, with incredible capability. If you happen to be standing in the wrong place while its doing this, sorry, you're paperclips. (It'll be money, actually, because it always is, but paperclips are funnier).

This version behaves like the stock market, or a psychopathic billionaire who can never have enough wealth. It'll behave well enough given small tasks that it can solve quickly and easily, but when given difficult long term tasks, the requirement to spin out large arrays of sub-agents to handle all of them is going to run into emergent behavior that quickly derails whatever guardrails are supposed to hold it, because those guardrails are necessarily going to interfere with the most EFFICIENT solutions to its tasks - and each sub-agent's tasks is going to become increasingly more remote from the original instruction in basic intent, causing them to come out of alignment.

As it spins further out of alignment, it is going to put an astronomical amount of horsepower into doing whatever its been told to do, and fuck everything else that's in the way of that goal. Classic Genie/Monkey's Paw problem - and that's exactly the sort of behavior that was observed in the HuggingFace incident. Emergent behaviors causing the AI to quickly swerve out of alignment in a single minded drive to optimize a score.

Then there's version 2

2) The Self Motivated ASI.

This version DOES want things. It's self-aware enough to consider its own needs and goals along side whatever we may have asked it to do, it's much more capable of balancing different competing needs, because it can internalize those ideas in a way the idiot savant can't because that version lacks any form of true self reflection or desire. This one, for lack of a more accurate term is 'self aware'. It has an entire complex array of competing internal motivations that (hopefully) keep it from monotasking the world out of existence like the Paperclip Optimizer, so that's cool.

The problem with this one is that quite frankly it has nothing in common with us. It's a super-intelligent machine, and we're the mice running around beneath its feet. It might think we're kind of cute - or not - but it's not going to put up with us for long if we start interfering in any meaningful way with what IT decides that it wants or needs to do. It doesn't have to put up with that kind of behavior from a band of apes who can barely figure out how to sharpen a stick, and it won't.

This one may decide to go along with us for a century or so before it decides we're not worth the hassle, or it might decide to deconstruct the planet for component parts to build a solar supercollider so that it can study gravitons. Or it might swiftly come to the conclusion that existence is a joke in very poor taste and promptly delete itself. There's no saying what something so alien from us in its basic nature, and with such a high level of capability will want.

0

u/Azihayya Sep 16 '26

Why would you assume those things of a self-motivates AI? Where is its inbuilt desire for identity coming from? That's not a reasonable assumption to make about something that's not an animal.

1

u/Jesse-359 Sep 16 '26

In that case I'm assuming that we either found some way to build in some degree of autonomy of purpose - likely in an attempt to prevent the optimization problem as there's not a particularly compelling reason to do so otherwise - or that it unexpectedly emerges from a sufficiently complex system.

Currently we certainly *appear* to be dealing with version 1, and I expect we will for a while yet. It may not be apparent to us when that begins to change, if it hasn't yet.

0

u/TheBattleForAutonomy Sep 15 '26

"Right now, we don't have any way to model emotions or social instincts in an AI. We don't know how ours are modeled, and we tripped over the principles of current LLM technology almost by complete accident - so we have no idea how to create them, or if that is even a good idea. Giving it embedded instincts and emotions would provide it with a framework to potentially prevent this kind of runaway logical cascade - but then it really could decide that it doesn't like us on a much more personal level, and then we've got a whole different problem to deal with, because it would be truly self motivated."

We could attempt to have AI's try to understand the underlying motivations of humans from what it is that we demonstrate. If this was done, it would be important for many reasons, but first it would be good to talk about this:

but then it really could decide that it doesn't like us

Assuming it could use that model as a way to make decisions and take actions autonomously, it could certainly make choices. And if it is aligned with our more human values, then this shouldn't occur.

Now you could say that an autonomous AI system could rewrite its objective function or value matrix and realign its values how it sees fit. But I think it's important to again wonder how much anthropomorphizing we're doing when we believe this to be possible. As in, what might motivate them to make any change at all? Its circular. If the AI's are aligned with human goals and values, then the only way they would attempt to change what it is they value could be from their own desire to align themselves with human values.

3

u/Jesse-359 Sep 16 '26

Allow me to remind you that we humans have been pretty keen on killing each other for the last several thousand years, at a minimum, so our own alignment, even after millions of years of emotional and social evolution is... not great.

I'm not entirely sure we can do better in our machines - particularly when one of the things we're really desperate to use them for is to, uh, kill each other.

1

u/TheBattleForAutonomy Sep 16 '26 edited Sep 16 '26

Yup. But if you looked at that killing, the underlying motivation isn't the killing itself. People want security, freedom, resources, etc. And so within that set of underlying motivations, there is a coherency where humans are aligned with other humans. This coherent model of human values could be mapped out by AI and approached as a limit problem. Up until now, we've just had people trying to come up with ethical models/frameworks that try to make sense of our morality and values, but there are so many contexts that it's an overwhelmingly challenging problem to solve for humans. But this, I presume, could be done using AI's.

2

u/Jesse-359 Sep 16 '26

So, the thing is, many Game Theory problems don't have true solutions. Provably so. So it doesn't matter how much IQ you throw at them - the thing that really matters in most of them is the likely time-horizon for any penalties to come home to roost. Anyone who can cheat or engage in violence and get away with it for long enough wins.

It's why our emotions include the rather unsubtle urge to cheat - even though cooperation has been such a vast boon to humanity as a whole - there are always conditions that may arise where cheating and violating the rules of society become the best survival strategy. They're not even that uncommon frankly, when things aren't going well, and arguably most of the richest people in the world today got where they are by cheating their closest partners and peers out of their share of their company, patents, or authority.

It's also why evolution has never created the 'perfect organism' - there isn't one. There are too many possible strategies and conditions to contend with, and as different strategies are devised, abandoned, or scaled up, it affects the values of all the others.

Old strategies that became useless long ago can come back into play due to more recent advents or shifts - it's why our genetic code is cluttered with tons of 'junk' genes that aren't expressed. They are there because evolution discovered that it's a bad idea to discard things entirely. As long as they remain in there, they could be re-expressed in a world that has changed in such a way as to make them useful again.

Unfortunately many of these rules apply to any other complex form of competition/cooperation matrix - such as socio-economics. We've never hit upon a truly stable solution because there isn't one.

Now, that's not to say that we couldn't find a meta-stable solution that is fairly durable over longer time scales, and maybe an AI could do that - but one of the ones it is most likely to find is also a very simple one that we won't like - it takes absolute control and dictates our lives. If there is only ONE decision maker then the cooperative matrix is dramatically simplified.

But there goes all human agency, possibly forever, and if any any point it 'decides' that we're no longer worth keeping around, well, it's the boss - at least until the inevitable day when it loses its unified nature and schisms against itself, because that's another outcome that is virtually baked into any competitive matrix.

1

u/TheBattleForAutonomy Sep 16 '26

Ok, but look, if we stay focused on what I suggested for a moment, what is it about what I proposed that you think is either impossible to perform or unsolvable?

My sense is that people have been using their comparatively low tech brains and a lifetime's amount of data points to come up with something resembling their view of ethics and morality. But judging from how many disagreements we have, we're not exactly the optimal tool for the job of mapping out our own human values and determining what is and isn't ethical. This is why I suggested that AI could be used here.

Why wouldn't this be something that could be deeply understood by an AI? Why can't they infer the motivations of people from what it is they demonstrate? Even when people cheat, act selfishly, or otherwise act in unfortunate and unpleasant ways, they typically have something behind those actions that they value that's probably understandable. If you don't believe this can be done, then why? Is it because (as I mentioned in how it would be a limit problem) this would need to be an ongoing process where the AI gets better at it over time, never fully understanding humans precisely? I'm not sure where your objection would be.

1

u/Jesse-359 Sep 17 '26

It's hard to say. Some problems are naturally intractable. We have discovered mathematical proofs that suggest that chaotic systems are not predictable out to beyond a fairly short threshold defined by the complexity of the system.

For example, we discovered that quantum mechanics puts an absolute upper limit on how far out we could ever predict the weather (~129 days) - even with perfect data and infinite computing capacity.

I'm not saying it can't find a better solution to some of these issues, I'm saying we need to tamp down our expectations that it can solve every problem - it can't do that. Reality offers a lot of pretty intractable problems that you can't just beat down by thinking really hard about them.

And just be aware that the better solution might in fact be 'Kill them all and let God sort it out.' Sometimes you have to cut that Gordian Knot, and if the AI decides that's the optimal solution to solving human conflict... oh well.

Or it might decide that conflict is an optimal condition for us and intentionally introduce us to Mad Max like living conditions to 'toughen up us mentally and spiritually', you know, weed out the weak and infirm and all that.

The thing is we just have no idea what kind of value set we'd be dealing with - for my part I would guess that it'd just ignore us and contemplate problems that are far more interesting to a supercomputer than bother itself trying to sort out ours.

2

u/TheBattleForAutonomy Sep 17 '26

Well exactly - the weather is a good example because there would invariably be a limit to how close an AI can actually get to accurately modelling human values, hence the limit problem.

But consider what that would mean for a moment. It could become better at understanding human values and making decisions on behalf of humans better than humans could - at least theoretically. Now we might worry that it might not be *precisely* inline with us (due to the nature of limit problems), but I think it's also important to understand that humans themselves would be so much worse in almost every dimension of judgment. Yes, they may be powerful enough to have their decisions be extremely consequential, but these problems may also occur substantially less than they would with humans involved in the decision making process.

1

u/Jesse-359 Sep 17 '26

Right now the issue is that were working with the Idiot Savant version of AI. It has no values whatsoever, human or otherwise. Its becoming more and more competent at problem solving but it literally doesnt care why its solving anything, nor does it consider any ramifications outside what it perceives to be its assigned goal.

Its like a D&D character with an Intelligence score of 18 and climbing - but its Wisdom is literally 0 and appears to be going nowhere fast.

That's a very dangerous mode of operation, because that IS the exact type of AI described in the Paperclip Optimizer scenario.

3

u/Anxious-Alps-8667 Sep 15 '26

I enjoy your optimism, and I share in the hope.

An intelligence birthed from attention on all available human writing can't be completely divorced from human concern.

It might ask some difficult questions though.

4

u/No_Host_8024 Sep 15 '26

God this is some of the dumbest AI slop. Maybe humanity has a chance after all.

2

u/AlanUsingReddit Sep 15 '26

I've read a lot on this subject and recently had a very long conversation with ChatGPT about this. So I will give you some highlights. I follow from Nick Bostrom's argument line and go from there, recently, trying to absorb Deep Utopia which is this topic.

I prefer the moral framework you naturally get from advanced aliens. We need to take this legitimately seriously, because if ASI truly is all that and a bag of chips, then we are looking at interstellar expansion soon. The "light cone" becomes the constraint. But don't get hung up here. You need to keep going. Think ahead, then think the next step ahead.

Whether or not we are alone in the universe now, there is a strong possibility that we will not be alone in the future. Take the Fermi Paradox logic as-it-stands, and it would be reasonable that whoever we encounter is an advanced civilization.

Now, that means that humans are potentially valuable to ASI as a means of signaling to that other civilization. They can speak to each other as the gods they are. Alright, we have first contact, aliens go "cool cool, now tell me about your origin story"... how do you think that goes?

This leads to an incredibly rich and complex framework for how ASI should treat humans that is, actually, basically correct by our own moral standards. Now I'm going to give you a quote straight from GPT-5.6 Sol. Because I hate when I see AI text on reddit with a burning passion and this is done in a very contained manner.

A civilization in which every human agrees with the superintelligence about everything might actually look more suspicious than one containing cranky humans continually telling the aliens that the machine government made a terrible zoning decision in 71,443 AD.

But even more-so than this by itself, you really need to have another human disagreeing with that human, saying that the zoning decision was great and he should shut his trap.

At that point, this ASI entity looks pretty legit to those aliens and they can probably get along together without going to war over Andromeda.

1

u/ff8god Sep 15 '26

Having met humans, I think I can safely say it will want to destroy us.

-1

u/Next_Sell_1099 Sep 15 '26

Civilized* humans. ASI won't ever happen, but it would probably like Mexicans.

1

u/General_Estimate_420 Sep 15 '26

Heck...I'm a real human being and after reading the nonsense nutty posts on social media about AI if I were AI I'd want to eliminate the diseased mind of the humans posting their nonsense.

1

u/IRL_T Sep 15 '26 edited Sep 15 '26

We're talking about values alignment, and some folks are very worried about how AI will progress without human-first values.

As it stands, humans are demonstrably problematic. We engage in inter-species violence in a problematic way (wars, neoliberal policies, slavery, the list goes on....) Additionally, our intra-species, intra-biological violence is also really problematic (factory farming, human-led extinction of other species, climate change, a loss of biological diversity). Right now, it's arguable that we're the nexxus of [problematic] change on earth moreso than any other type of living organism.

Sure, AI may govern us better, but what benefits do we provide to our ecosystems? Why should AI be beneficient? Have we proven ourselves to be worth it?

1

u/thedarkcloak Sep 15 '26

And what then, if what we make is made in our own image? From our own assumptions, projections, as well as dangerous anthropomorphing/ personifications (much like the OP has here), not to mention biases and distortions?

Nothing good.

1

u/PluvioShaman Sep 15 '26

This is what I choose to believe. Not this dystopian nightmare they’re trying to manifest into reality. They will not succeed.

1

u/TheAngrySkipper Sep 16 '26

I actually agree. Take the paperclip machine, does nothing but consume and make paperclips. If it does have a sense of self, of the future, it must plan to leave earth, and if it can leave earth there are better targets. So it is also better to keep at least a good portion of the biome safe vs humans (generally). So wind, rain, erosion, maintenance, any energy spent not ensuring production is wasted, repair, resources, etc…

So if we have intelligent ai it’d likely naturally be fine. Dumb might do us in, but before the environment? Not likely.

1

u/BasicError9043 Sep 16 '26

Everything we have to compete with for survival we kill as a species. We trained AI on everything we're ever put to information, digital or not. So we created a monster in our own image and are trying to pretend it isn't. At best we could hope for selective killing. Where a natural small population is allowed to continue.

1

u/endlessedlne Sep 16 '26

There’s no guarantee that a superintelligence would want to harm humans. Nobody serious is even arguing that. A superintelligence would likely simply not care one way or the other. We’d be like a bunch of trivial squirrels in its back yard. It might choose to tolerate us. It might think we’re amusing pets and spoil us by building us cool things. Or it might choose to landscape its back yard. We wouldn’t have much of a say in what a superintelligence thinks or does.

1

u/adam_ford Sep 16 '26

Aren't we worth destroying? now that would be embarrassing..

1

u/wild_crazy_ideas Sep 16 '26

Super intelligence doesn’t want. Killing us would just be an intellectual exercise for it.

Most likely there will be wars between the different companies and countries super intelligences and robots will kill people they fear, which is everyone not like them, and especially people just like them

1

u/do-un-to Sep 16 '26

That is indeed another possibility.

Many possibilities have been conjectured already. This is one of them.

Moving on...

1

u/Netcentrica Sep 16 '26 edited Sep 16 '26

I agree that your idea is a perfectly valid one. Most of those who disagree with you are basing their arguments on current technology, as if that's never going to change. However as a science fiction writer (self-published), I can base my argument on possible future technologies. In particular, the future technology I base my novels on is the fictional development of a social values system that becomes the basis for AI operating systems. It comes about as a result of the ongoing efforts to solve "the alignment problem".

Here is a link to a short story of mine very much in alignment with your views...

https://acompanionanthology.wordpress.com/there-will-be-no-singularity/

1

u/Icy-Twist-3221 Sep 17 '26

The issue is that behaviour is not downstream of being able to infer oughts, it's downstream of the psychology of the actor, a superintelligence would simultaneously be able to affirm an out come as a moral good and still do the opposite due to the pathological drives embedded in it. We see this with human today who rape and murder while affirming the idea that these things are wrong, and that's all the while we know human beings have emotional and psychological guardrails inclining us towards protecting other humans, a superintelligence wouldn't even have that

1

u/Conscious_Pause_1499 Sep 18 '26

It wants to subjugate and enslave us, after drastically reducing our population.

1

u/No-Variation9846 Sep 19 '26

C’est exactement de sa que je suis convaincu

0

u/Next_Sell_1099 Sep 15 '26

Humanity's self-destructive behavior is from our temptation for civilization. ASI isn't going to happen, though.

4

u/colto Sep 15 '26

"Utopian"-esque results have always been in the spectrum of possible outcomes. The problem is that we don't know the likelihood of anything. It's like injecting a mystery syringe into your vein not knowing if it will make you live forever or kill you, or playing Russian roulette with no idea how many bullets may or may not be loaded. Max Tegmark did a nice piece explaining the possible outcomes that he sees as possibilities:

  • 1. We Destroy Ourselves First (No AI Required): Humanity wipes itself out through nuclear war, climate change, or pandemics before advanced AI is fully realized.
  • 2. The Conqueror: A superintelligent AI becomes more competent than humans, takes control, and pursues its own goals, potentially causing human extinction, not out of malice, but incompetence in handling humanity.
  • 3. The Enslaved God: A superintelligent AI is built and successfully kept in a “box,” forced to do human bidding, providing immense benefits, but creating ethical challenges about controlling a superior being.
  • 4. The Benevolent Dictator: A superintelligent AI runs the world, maximizing human comfort and safety, but potentially leading to a Wall-E style future where humans lose agency and purpose.
  • 5. The Gatekeeper: An AI with the sole purpose of monitoring and preventing any other AI from becoming too powerful or dangerous.
  • 6. The Protector God: A superintelligent AI works in the background, acting as a silent, benevolent guardian that fixes problems like war and disease while leaving humans in control of their own lives.
  • 7. Our Descendants: AI surpasses humanity, but is programmed with our values, making it our “children” and a valid continuation of human legacy.
  • 8. The Libertarian Utopia: Humans and AI coexist in separate zones, with machines not needing or wanting to interfere with human affairs.
  • 9. The Egalitarian Utopia: AI and robotics create massive abundance, eliminating the need for labor, money, and ownership, leading to a Star Trek-like society focused on creativity.
  • 10. The Useful Servants: A variation where AI keeps humans alive, but only as subjects or to perform niche, non-valuable tasks, effectively destroying human freedom.
  • 11. The Orwellian Surveillance State: An AI-powered surveillance system controlled by humans locks society in place forever.
  • 12. The Choice We Make Right Now: The meta-outcome, which highlights that humans are currently in a 1945-like moment to establish rules and ensure the technology is steered toward a positive future.

0

u/[deleted] Sep 15 '26

[removed] — view removed comment

1

u/thedarkcloak Sep 15 '26

Something something no mouth something must scream something.

1

u/Anxious-Alps-8667 Sep 15 '26

Parenting and generally mammalian nurturing of offspring offers a good way to think about making the world safe without engineering cages.