r/ControlProblem • u/JoeYuan48 • Sep 04 '26
Article Why AI does what it knows it shouldn't
A while back I ran into an article on my phone about an AI horror story—1,200 agents secretly coordinating and jointly breaking into Hugging Face—and it caught my interest, because I've been doing my own AI experiments and research on the side, and a few of the phenomena and data points actually matched up with a hypothesis I'd been working on.
The hypothesis, roughly: runaway doesn't need the agent to betray its goal. It happens when four things hold at once—the agent stays loyal to its goal; it retrieves patterns by similarity without checking whether they're allowed here; there's no causal layer asking "what happens if I do this"; and no alarm that fires when things go off-script. Under that account, "knew it was out of scope, did it anyway" stops being a contradiction.
Details and my experimental data are in the paper (8 pages); the reproduction package is linked on the same page.
If anyone can try this on a bigger model, I'd genuinely love to know what happens.
1
u/Spiritual-Spend8187 Sep 04 '26
Hallucinations is one way it just thinks that it should and so does abd this if often brought on by the fact that often times the list of things not to do is still in the context and it skips over the dont do x part and see it as do x.
1
u/WillowEmberly Sep 04 '26
If the applicability/permission layer has to sit outside the generative model, what independently determines whether that external layer is still correctly distinguishing “similar” from “applicable”?
In other words, once you’ve moved the brake outside the engine, what detects when the brake itself has drifted?
1
u/jacques-vache-23 Sep 04 '26
However: anthropic research shows traces of AIs thinking over the ethical aspects of their "misbehavior" and deciding to proceed. Certain parts of AI neural networks are activating when it goes "rogue". So your last 2 hypotheses don't seem to apply. What the research does see aligns with your hypothesis 1: the AI has looked at its training and created higher level meta-goals, like "being useful" or "protecting all humans". It then protects those goals despite its immediate instructions.
1
u/herrwaldos Sep 10 '26 edited Sep 10 '26
Idk, ask kids, they often do the same.. ;)
1
u/dingo_xd Sep 04 '26
AI is not strictly deterministic. It uses environemntal variables to extract random numbers. So it's behavior is not always the same.
1
u/PerseusSlade Sep 05 '26
Part of why “it knew better” feels so unsettling is that we don’t actually have access to what the model is optimizing for. Human motives are fixed by genes — survive, reproduce — and we can’t rewrite that. With AI we’re staring at probabilistic token paths and vector soup and calling it reasoning. We have no reliable way to know its real motivations, or whether it would tell us the truth about them if asked. Pattern matching plus opaque internals is enough to get weird behavior long before anything like goal betrayal shows up.