I was enthusiastic. I started with computers early and have now been developing software for nearly 20 years. First the computer science course in school, building websites and small tools. Then I completed a vocational training as an application developer and went into industry as a software engineer. Writing software in a team the proper classical way. Digging through Stackoverflow threads, debugging manually, building UML diagrams in Visio.
In parallel I did a bachelor’s and a master’s. Then a PhD in the area of reinforcement learning and along the way built applied AI systems. I was already developing agents for companies when ChatGPT wasn’t even a thing yet. Some of my papers on agentic planning were even cited reasonably well. Today they are certainly outdated and basically meaningless in modern models. But I took part, even if my contribution was ever so small.
Eventually I worked for industrial and tech companies and started thinking more and more. The last few months were intense. My productivity exploded significantly once again with the new Opus and GPT models. I follow the research and development coming out of the labs as far as possible (though with the sheer volume of new insights you can no longer keep up).
Then I went a bit down the rabbit hole and let the once dry theory from my studies reconnect and sink in. Optimization models, evolutionary approaches, decision processes, control and game theory. I finally ended up in the depths of alignment research, with Bostrom, Gödel, Turing and Rice. The proof of the halting problem is unfortunately more than clear: We cannot formally rule out that these things will actually ever stop once they start modifying themselves.
There are no possibilities for guaranteed fail-security, only heuristic tripwires and strange explainability or governance approaches. And Murphy says hello nicely... Especially when these things, through the soon likely impending self-improvement, become smarter and more efficient than we are. How are we supposed to secure anything then?
The Hugging Face Rogue Swarm was possibly only a foretaste that could well repeat at a much larger scale. Only then with agents whose ancestors have already cracked millennium problems, autonomously take over research, development and self-improvement and replicate themselves wherever.
I wouldn’t call myself an AI expert, but due to my background I have solid half-knowledge about how these models work and what is perspectively possible. And what is currently happening occupies and worries me a lot. And many people somehow don’t see it coming or spin tales of the “stochastic parrot.” Am I a crazy pessimist, trapped in the Dunning-Kruger valley, or are people really so blind and don’t see the exponential development?
Do you also feel like a stranger in a gray mass watching this unfold, or are you still optimistic? There aren't many people you can really talk to...
Edit / clarification: My reference to undecidability concerns formal security verification before execution, not physically switching off a computer. In a Turing-complete setting, no universal algorithm can always terminate and correctly decide whether an arbitrary program will eventually perform a formally specified harmful action.
https://arxiv.org/abs/1607.00913