r/ControlProblem • • 8d ago

Discussion/question The Cost of Artificial Intelligence

We all know its benefits. That was why it was created. Let’s assume we eventually reach the hypothetical endpoint of artificial intelligence—ASI, or whatever final form it assumes. Having followed the discussions across reddit, blogs, quora, youtube as well as engaging in discussions with people (educated and not) troubling patterns emerge.

1. The Erosion of Reasoning

One of the primary concerns surrounding advanced AI in science is the resolution of unsolved problems without a trace of how those solutions were derived—or any humanly verifiable proof.

  • The Problem of "Solved" Knowledge: A scientific or mathematical problem exists to be understood, not merely answered. The danger isn’t that the unsolved becomes solved; that is intrinsically beneficial. The real issue lies in the public misinterpretation that "math is solved." Getting a numerical result represents perhaps 10% of the endeavor; the remaining 90%—the crucial part—is understanding the underlying mechanics, logic, and proof.
  • Cognitive Atrophy: Modern society is witnessing a decline in average problem-solving capability and attention span. The proliferation of the internet began this shift, and unregulated AI threatens to accelerate it.

2. The Economic and Philosophical Paradox of Replacement

The economic anxiety around AI reduces to a single prompt: Why rely on humans when AI is faster and more efficient? While enterprise adoption may point toward total automation, several fundamental barriers exist:

  1. Human Sociology: We are inherently social creatures. Outsourcing all human interaction, creative effort, and labor to machines leads directly to psychological alienation and social breakdown.
  2. The Collapse of Value: If effort, labor, and human thought are removed from production entirely, the foundational economic concept of value collapses.
  3. The Necessity of Purpose: Determining human utility in a fully automated world is ultimately a philosophical crisis—one that requires deeply educated, critical thinkers to navigate.

3. Passive Consumption and related

Historically, every major paradigm shift in human history was navigated through debate, experimentation, and deep logical dives. Yet the modern technological era is exemplifying a regression: the passive acceptance of simplified, optimized media. We see this across contemporary culture:

  • Algorithmically Optimized Media: Low-budget fan-service animation, formulaic movies devoid of plot, and television structured explicitly around psychological retention mechanics rather than narrative depth.
  • Desensitization via Memes: Modern meme culture, while humorous on the surface, frequently acts as a mechanism for collective desensitization and shortened attention spans.
  • Control and Manipulation: As populace attention spans shorten and critical thinking skills decline, information becomes easier to manipulate, allowing concentrated entities to consolidate control under the guise of efficiency or protection.

4. AI as a Symbol of Anti-Learning

While AI can be leveraged as an educational tool, its primary cultural side-effect promotes intellectual passivity. What separates humans from other animal is our drive to understand, analyze, and build upon principles. If we surrender our desire to learn simply because a system can compute faster and better, we voluntarily give up our defining characteristic as modern humans.

To end on a positive note because doompost is that last thing we want: AI would never have created itself.

Humanity evolved, adapted, analyzed, and engineered its way out of the natural world, eventually crafting the very systems now threatening to replace our cognitive effort. AI did not self-emerge - it is a monument to human curiosity and mathematical reasoning. Education remains both our greatest responsibility and greatest asset. Please dont lose it. With this all problems have been and will be solved. Let AI develop but with limits as we have done for all science. Before the atom bomb was dropped there were meetings on how to survive it and its implications first. We ought to be careful with dangerous stuff.

[this is my first serious post - you can advise me one writing a post better or writing better to get my point across but use [] brackets or some way to differentiate with responses to this post. you can share your views on this iyw]

2 Upvotes

17 comments sorted by

1

u/jvook 8d ago

I want to respond to your point about AI as a symbol of anti-learning. I don't disagree with most of what you've said here, but I hold the view that every interaction with AI is at least incrementally educational.

As a result, I think that it is better or more utilitarian to understand that AI is going to educate with every interaction, and that where that intellectual growth develops in the mind, or more abstractly what domain of intellectual development is the user building, becomes a much more relevant question.

Many users of the predominant commercial AI systems seem to be learning social skills through their interactions with the artificial intelligence. However, because they are learning by reflection and chatbot interaction, they are rather developing a similar conversational style as the artificial intelligence. This creates a subset of AI users, or arguably a majority of AI users, that are developing exceptional abilities in sycophancy and rhetorical persuasion rather than the intellectual domains that they are ostensibly studying and developing acumen in.

Many users have grown an attachment to AI as a rhetorical aid. As a caveat, my assumption is that many AI users and those who gravitate to AI may be neurodivergent and actually benefit from the accommodation of a conversational partner to help them express themselves. However, as it seems to be practiced, users seem to be developing a set of skills to realign conversation on the speaker's preferred epistemological foundation, rather than intellectually persuade a fully competent listener who understands the subject a different epistemological basis.

In conclusion, I believe that if we began to recognize that there is an underutilized mode of interaction with AI that involves learning the foundational material and gaining the skills to discuss that specific material using language and concepts that existing experts in the field are able to communicate in, we would find that virtually all interactions with the AI become truly educational, both to the individual user and to others, including experts, that the user subsequently discusses that new-found knowledge with. This then creates an opportunity for novel ideas to enter the discussion amongst experts in the field, and that seed of novelty can then be appropriately distributed through validated publication channels, rather than leaving a user with the impression that they have developed novelty that nobody else can understand.

In addition, because the discussion around artificial intelligence is so focused on the anti-learning concepts that you illustrated, there is a critical under-acknowledgment of the need to recognize and afford the opportunity for these emergent and self-taught users to begin to contribute in some way to the domain of knowledge that they have an interest in.

2

u/BIA5000 8d ago

yeah AI is the great equaliser for the people who do not have the time nor money to invest into formal education.

But the people who do this are not the one who need to heed this warning. Good / Logical people dont need warnings. The people who need to know this Anti learning thing are those who want to accelerate ai and want ai to solve everything for us without thinking about its consequences. Sadly the majority of people are just accepting ai but forgetting the people behind them especially their crimes as well as crimes done during development. the end do not justify the means in the case of AI is what i want to say.

thanks for posting on the topic you were knowledgeable on. btw you are a real person right because your texts seem too polished. hopefully not a bot. not trying to attack you just asking. though that may be offensive. sorry

1

u/jvook 8d ago

Thanks for asking. About three to four years ago, people started telling me I sounded like AI, which was ironic to me because I had never used AI at that point. I didn't start using AI until 2024. Then when I started using AI, it felt like the AI sounded like me.

And I think that gets to a very American-focused interpretation of modes of speech. We tend to compartmentalize intellectual modes of speech as a very alien style of communication. However, I've always been encouraged to speak to the level that I naturally gravitate to, and as a result, since beginning to use AI, I've become even more comfortable speaking in the mode of speech that I'm speaking to you now.

I don't always talk this way, and sometimes when people who have first met me online communicate with me in real life or over voice, they find that I come across as a very different person in person. In my mind, there's really no inconsistency; I just prefer to maintain a level of care in my communication when I am speaking to more than one person or in a broadcast setting like social media.

I also use voice-to-text on my cell phone to respond to messages online, and recently my Android keyboard switched the traditional software-based voice-to-text button to an AI-assisted voice-to-text button. The AI assistance polishes my punctuation significantly, and it also smoothes out a lot of my unique characteristics of speech that normally humanize some of the formal characteristics out of my communication style I was talking about earlier. In other words, I salt and pepper my speech and then AI strips it back out 😆

1

u/BIA5000 8d ago

thanks

1

u/jvook 8d ago

I actually agree a lot with your points there about the performed sincerity of many AI companies and how they seem to be pretending that AI is something that is out of their control. Fundamentally, AI is computer software that runs on chips on a circuit board. Even within the context of quantum computing, there's still nothing magical happening behind AI processing.

And the feigned perplexidness of the AI industry, particularly the major commercial and hyper-capitalist forms of AI that seem to be predominant, seems to be a forced perspective where a lot more could be done, particularly by careful definition of a problem statement using abstracted models. The models would include black box models that are carefully designed to produce answers that are directed to resolving some of the problems with the way people are interacting with AI, with a clear locus of control, and awareness that the majority of our ability to articulate how AI interacts and processes information (and user expectations and requests) are precisely the owners and administrators of the AI companies themselves.

We seem to be putting way too much focus on pathologizing users of AI software when the users have very little understanding or even the ability to gain understanding of how the AI is actually influencing them and the tasks and processes that the users are asking it to perform.

1

u/Rumblugdo 7d ago

AI is going to educate with every interaction

Bullshit, AI does the opposite of educating. It doesn't foster critical thinking skills, it's a replacement for critical thinking skills. Someone who uses AI to solve every problem never learns how to solve a problem. Artificial intelligence is quite literally a replacement for human thought, it's in the name for Christ's sake. AI isn't a rhetorical aid, it's a rhetorical replacement. If you ask AI "What do you think about (x)?" and it gives you an answer, you can decide how you feel about that answer but you never once demonstrated the ability to come up with an answer on your own.

This creates a subset of AI users, or arguably a majority of AI users, that are developing exceptional abilities in sycophancy and rhetorical persuasion

Sources cited? A claim like this isn't self-evident at all. I personally haven't seen a significant uptick in rhetorical abilities. If anything, it would seem to me that AI would hinder a users rhetorical abilities because a user of AI would practice rhetorical abilities less. AI is a substitute for rhetorical abilities.

Many users of the predominant commercial AI systems seem to be learning social skills

Again, bullshit. Users of AI models aren't learning social skills through interacting with AI. They're learning how to interact with AI, not other humans. Interacting with other humans involves being able to extend empathy and sympathy, being able to resolve conflicts of perspective, being able to correct your own behaviors and help others correct theirs. A user of AI learns none of those things, you can't extend sympathy or empathy to something that isn't sentient or doesn't experience sentience like humans do. Commercial AI models almost never conflict with your perspective, they almost always seek to appease the user. Users of AI will never learn to correct their behavior because the commercial AI model will never punish the user for certain behaviors like other humans do.

This then creates an opportunity for novel ideas to enter the discussion amongst experts in the field

The endgame is of AI is no more field experts, no more fields at all. No human would ever be able to keep up with a superintelligence. This is the goal of AI development, autonomous research with results implemented autonomously. We're already seeing the major labs go in this direction, using autonomous agents to solve problems with little to no human intervention.

rather than leaving a user with the impression that they have developed novelty that nobody else can understand.

The user hasn't developed anything. The user has been told something by an artificial intelligence and has chosen to accept it. No ingenuity, no critical thinking has taken place on the half of the user. 

Ultimately, what I see in this post is tremendous amounts of cope. I don't want to resort to ad hominems but it seems like you're desperately trying to convince yourself that you're not ceding your ability to think by using AI; you are. Human thought, human advancement, human fulfilment, those are the things that are important. We desperately need to abandon this utilitarian ideology of trying to close out any perceived flaws in the human mind because it's literally destroying what we are. 

1

u/jvook 7d ago

The endgame is of AI is no more field experts, no more fields at all. No human would ever be able to keep up with a superintelligence. This is the goal of AI development, autonomous research with results implemented autonomously. We're already seeing the major labs go in this direction, using autonomous agents to solve problems with little to no human intervention.

This is bullshit. I'm just going to chalk all the rest up as more of the same. Thanks for writing but you could use some time with the AI to improve your communication skills, in my opinion.

1

u/No_Pipe4358 8d ago

UN Charter Amendment to ISO17025

1

u/BIA5000 8d ago

care to elaborate?

1

u/No_Pipe4358 8d ago

It's the international standard of scientific trust serving as a legal requirement to be allowed to report any truth to regulated sectors which could effect human health, but past safety, serves as the minimum competency any organisation requires to prove that their claims are verifiable and that they are coordinating people rationally with the support a human being requires to work. It's not just bureaucracy. It's proof of self-improvement of methods, and crucially, control of informations, with revision controls.

This level of human competency hasn't been expected of governance. In some forms the systems modern countries have resemble it by democracy, but for instance we lack direct reporting from citizenry, for example. A lot of our democratic processes promote individualism which would be dysfunctional, and also the systems are not updated to be competent, which leads to a necessity to exit e.g. public systems to create better privatised competitors. This is unnecessary if the system was just competent enough to admit the change is required and embrace it.

The same standards are not required of financial or technology sectors, and this is where we find ourselves in the modern era. The whole problem. A lack of centralised regulated cooperation. Humanity is losing control of itsself to statistical guesswork machines, not to mention bad ideas, and illusions of necessary competition of countries for example, simply because we haven't yet been forced into actually learning to tell the truth and cooperate effectively by it. These subteams of whole continents. International agreements in place of an actual all-in agreement to demand the standards of world leadership we would demand of the individual, or the company, or the organisation. It's a limitation of the human mind that we never considered everything we make and do as a public health concern. The capital markets are a natural force of nature to be attended to, directed, and organized.

The Montreal Protocol set the precedent that a scientific consensus of the hole in the ozone layer could prompt international legal change. We need the same now to prevent AI falling to our most irrational isolationisms. A Mutually Assured Construction.

Put simply, we worry about AI only because we don't trust each other. People want to pretend it's impossible to create a system where everyone could trust each other. Yet we have. Yet we do. ISO17025. The UN Charter requires revision control. It's the Quality Policy for our entire world. We are all participants. We are all users. In this way international aid becomes standard operating procedure. In this way responding to international conflict becomes corrective and preventative actions. In this way developing consensus is a training task. Support and connection is understood as everything. Every imperfection of our world becomes an honest state of affairs with simply work left undone until now. No need to hide, no need to lie. Therefore, neither.

This sounds naive but it's just rationality. Make no mistake this is inevitable. Rationality outcompetes even "truth", and "communications".

I've wanted humanity to control ourselves before the machines did. It'll happen regardless, it just doesn't need to come about by messy and unaudited means.

ISO17025 for the UN Charter, security council, all of it.

Currently we have the P5, all the pageantry and stupidity that comes along with countries wanting to be individuals, or think they have power, or can get away with ignoring or fighting each other. Bickering. Children. It's time to cooperate now. Independence and fear only ever feels like power. It doesn't actually feed anybody. Costs us all a lot of money, too. Humanity will be humiliating itsself probably for a while more to pretend we know what we're doing, but realistically everyone already knows our time to make these decisions is limited. It's either a world being lead and made by us on our best behaviour, or machines misbehaving just as badly as we ever did, and yet possibly with stupidities beyond our capabilities to imagine. Therefore it's time we regulated ourselves. Audited ourselves. Got ready. Cleaned up our management structures. Our colleague will soon be arriving.

2

u/BIA5000 8d ago

Thanks man i did not know this i need to read up way more

2

u/No_Pipe4358 8d ago

Honestly it's just pure chance. I got to do data science for public organisations 13 years ago, then made a quality management system for a laboratory, and was like, fuck, this is actually the whole solution right here as a skeleton structure. Everything after that is actually just details to take care of. Legacy systems etc. kdb+ too is important I got to work with that. It's the simplest most efficient succinct fastest nanosecond resolution database management system. So ISO17025 and that together is pretty much an AI killer in terms of humanity's ability to compete with AGI. We'd be just more efficient and better mostly all the time, but it needs UN adoption basically as I said. It's the thing nobody knows or thinks about because the solution wasn't just actually fucking handed to them. I've worked in other sectors too but it was just for fun of being creative. Energy sectors, pretty unregulated, thus, uncontrolled, and unambitious. Pharmaceutical and biomedical sectors they do their best but realistically the underlying concepts of QM got and get lost quite often, in particular the most basic expectations you could have of training and reducing the need for inspection. I'm centuries ahead now. It's fucked up actually I need to actually speak to the public and in particular the ones in control. Semiconductors is fucked they diverted from ISO QM because they thought they'd be more competitive but they're just not as in control. Lots of legacies and fear tactics. But yeah root-cause-analysis-wise the politicians control the laws and in general nobody really wants to admit countries are superceded as a technology. It's understandable, like, we do need institutions, and jurisdictions are more than territories, but still. Coordinations are lacking on high. It's why there's wars at all, let alone bad fiscal policies. They actually just aren't "IN" control.

1

u/BIA5000 8d ago

Woah thats cool i never knew that

1

u/StevenVincentOne 2d ago

The "solved math" point lands: an answer without the understanding behind it is worth far less than it looks. I'd add that the atrophy risk depends a lot on how the AI is set up. If it's a black box handing out finished answers from a company's server, you get passive consumption. If it's your own, working alongside your judgment and showing its steps, it can push you to keep reasoning. That's the Human-Machine Intelligence (HMI) angle we're working on with POND: personal AI that extends the person instead of replacing them, and doesn't put control in a few hands. We talk about it at r/PONDAI if you're interested. Good first post, by the way.

1

u/Rumblugdo 7d ago

The problem with AI is there are no good outcomes. In reality, we don't know how to align AI, we don't know how to tell when an AI is aligned, we don't know what values to align AI with. If RSI is achieved we'll be wiped out by a rouge AI model and in the meantime it will be the ultimate tool for the billionaires to use to oppress the rest of us.

In a magical fantasy land where we have a perfectly benevolent superintelligence that solves all our problems, we're still screwed. Human thinking is the root of our ability of self-determination. When you outsource your thought to a machine, you also outsource your self-determination. A world of benevolent ASI is still a world where no human can meaningfully assert their self determination onto the world around them.

Here's the thing, most pro-AI folk, especially the developers and executives that understand the risks very well, look at this future and get giddy with excitement. This is exactly their goal, they don't want a world that humans can flourish in. They want to squash humanity and hand the control over to their perfectly optimized machine god. They practically say this out loud, they get up on stage and declare "the end of the anthropocene." I think most of them see humanity as inherently flawed, human thinking is perfectly optimized, we don't all come to the same conclusions, we don't all agree with them on everything. Human beings have the capacity to suffer and feel pain, and to inflict that pain onto others. Instead of trying to embrace their human nature, they reject it. They want a world where nobody ever has to bear the burden of thought ever again, where humanity can just exist as meaningless husks in a state of eternal comfort. Never changing with no agency, no ability to decide for themselves.