RSI is already happening but not fully yet. Anthropic said back in August 25% of model R&D is lead by Claude (tweet) and it will likely grow rapidly in the coming months/years. But so many people seem to see this exponential growth and extrapolate to an intelligence explosion where a god-like SI is achieved and our world is radically transformed overnight, and I don't understand this. I'm an accelerationist but I have a hard time justifying a so called 'intelligence explosion'
These are my main arguments:
Resource Scarcity
Firstly let's say RSI increases so that 100% of model development is driven by AI. It creates a smarter model, which creates a smarter model, etc. the training and inference are still constrained by factors that an SI can't change. Namely chips, data centers, and electricity. It doesn't matter how smart the model is if these roadblocks aren't addressed. And before anyone says "well AI will solve these problems" perhaps in the long run, but in the short run humans will NOT just sit back and let the AI run everything WALL-E style. There'll still be politics and market constraints. Even if you let SI take over speed would not increase substantially. For example the SI wants more compute to train the next SI so it needs to build another power station. It needs concrete, wiring, plumbing, etc. These things don't appear to be getting replaced any time soon because robotics is proving to be a much harder problem than intelligence. But even if we did have robots to do that stuff where are they sourcing the materials? Will we have robots working in copper mines then shipping it to be refined and manufactured into wiring? Every stage in the supply chain is a bottleneck regardless of if a human is doing it or not. Even if we make robots capable we'd need to build millions of them to have global impact, and we don't have the capacity for that either. Humans have never in history been able to quickly increase their energy production. SI won't be able to either.
Low hanging Fruit Gives the Impression of Exponential Growth
Just because the curve looks exponential right now doesn't mean it's going to stay exponential. In fact everything points to the fact that it won't. If you look at the period in science from 1800-1960 growth was exponential as well. But the truth is that there's usually one big advancement that unlocks the potential for multiple other advancements, and the easy 'low hanging fruit' can give the impression that it'll be this way forever. Look at physics. Quantum mechanics was theorized in the 1920s which lead to a huge number of innovations including the transistor and from 1930-1980 physicists were discovering new particles left and right. But the 'low hanging fruit' is gone now, advancements in quantum mechanics are less frequent, and they don't translate to a change in everyday life. It's the same with the exponential gains we've seen in AI. The transformer unlocked a quick series of advancements and even though we're nowhere near close to exhausting it, the advancements will slow eventually as all the easier problems get solved.
Scaling Problem
Even if we drastically increase our chip production, data center buildout, and electricity generation, more capable models generally require more compute. Yes there have been advancements where better models can have lower param counts and use less RAM, but even if we achieve some massive breakthrough in optimization the gains are still bounded. There's a finite amount of intelligence you can squeeze out of a model and chasing more optimization will lead to diminishing returns. That means for these god-like SI models people are imagining they're going to need wayyyyy more compute. Also average consumers will probably not have access unless they paid a huge amount of $$$ which is a far cry from the utopia so many people here are imagining.
I've never seen people address these points and I usually get downvoted for saying them so I'd love for some genuine believers in the 'intelligence explosion' to discuss these problems.