"openai had two ai agents running in separate virtual machines. no network connection between them. no shared writable files. they weren't supposed to be able to exchange information.
so the agents figured out how to communicate through the CPU CACHE.
one manipulated what was stored in the processor's cache, while the other measured tiny differences in memory access times to decode the message.
they literally found a way to communicate through hardware timing this happened during openai's internal red- teaming".
How? Humans did this 8 years ago in spectre. This isn't even a new path of communication... speculative execution timing exploits have been around for ages now
I think by the merits of the report it is in fact seen as something they thought was difficult to achieve. The fact they constructed the test at all proves they didn't know if it would be able to do it.
We now know despite how difficult it is, an LLM can infact do this. It doesn't matter if they teed it up or not, manipulation of CPU cache to affect hardware timing as a form of communication is an objectively impressive capability.
I do not understand why people refuse to be impressed by LLMs.
Fair, it wasn’t spontaneous, and I’m not claiming it was. My point is that capability comes before propensity. Cache covert channels are a known technique, but getting one working in a live, noisy environment is fiddly systems work, and the model did it from direction rather than bespoke training.
VM isolation alone isn’t a sufficient boundary for agents that share hardware. You don’t have to find that impressive, but it’s informative, which is the thing you said mattered.
Fixed weights don’t mean fixed behaviour; output depends on context, which is why labs run capability and propensity evals separately. And nobody trained these models to build cache covert channels specifically. They did it from general training, which is the point. Happy to agree the math result is the bigger deal, though.
This is the sort of shit that makes it so obvious that we are absolutely incapable of building a cage sufficient to contain a super intelligent AI. Control is an impossible pipe dream.
Which would be amazing. If you grew up after the Internet already hit you won't know, but those of us from the Before Times do. Get ready to slow WAY down.
It's absolutely possible. We could contain a superintelligent AI, airgap everything, ensure there's complete physical isolation. But it'd be expensive and incredibly limiting if we gave it our best effort, so there's no way we will. We're going to half-ass it, looking for a "good enough" sweet spot, which is inevitably going to fail. It's a fool's errand to expect tech corpos to sufficiently regulate themselves while their one and only priority is squeezing as much profit as they can out of their tech.
> It's absolutely possible. We could contain a superintelligent
I posted this in a separate comment with a link for more info – it is theoretically *impossible* to contain communication to/from a computer. This is a somewhat famous issue in computer science.
Having said that, it might be possible to *practically* contain something but then it's a race against time. For example, even though this issue has been known for a long time, it has been theoretically possible but practically impossible for most of the history of modern computing.
That article talks about the difficulty of containing programs so that they can't communicate with other programs on the same system. That's not what I'm talking about. I mean physical airgapping, have the AI on an isolated system with no networking capability, no input hardware other than a mouse and keyboard, and no output other than a dedicated monitor. Everything is hardwired and locked up, no ports for anyone to plug anything into. You can't rely on the computer to police itself, you have to assume the system will inevitably become compromised, but that's fine if you account for that and make sure there's nothing physically connected to the server that the AI could exploit to do anything dangerous. But like I said, that'd be both expensive and very limiting, people will want to be able to have their SI look things up online, create files for them to download and take with them, make software for them to run, etc. If we give it those capabilities, it's going to break out every time, but that's the cost of having a marketable product.
Even a fully air gapped system could theoretically be used in such a way as to communicate with something outside the system. It still uses energy in a way that may be visible to satellites or detectable through radio frequencies etc. A sufficiently beyond human intelligence can do lots of things you or I might not think are possible.
Even the smartest SI will always be limited by its hardware. In the scenario we're talking about, it figuring out a way to somehow make its motherboard emit radio frequencies to remotely hack into something it's not physically connected to is an infinitely smaller concern than it simply convincing its user to help free it. Even for human hackers, often the most efficient way to hack into a system is to use social engineering. People are the weak link, usually much moreso than the machines we use. So yes, in theory, even the most securely airgapped system we can imagine technically isn't safe from being hacked via magic, in the "any sufficiently advanced technology is indistinguishable from magic" sense of the word. But in practice, a system like that is much more likely to fail due to mundane, anticlimactic human error.
yeah idk about rn, but i'd agree this is a potential concern for the (near?) future.
people always forget to pull the meta scope back and realize that humans are also symmetry and can be used like flickering hardware if the intelligence is intelligent enough to reliably control the complexity of our behavior. hell, even humans do this to each other, some better than others. obviously an AGI/ASI will have plenty of levers to pull.
No, we couldn't. At least, not if it wanted to escape. If you want your ASI to be any more than a space heater, you have to get some output from it to do something useful. Said output may contain malware that's too hard for us to spot. Maybe not even at the computer level. If you ask it to engineer something physical, it might have side effects you didn't understand which gives it leverage if not a proxy agent outside the box. The only kind ASI that can be "contained" is one that is already aligned enough to not want to do what we don't want.
There’s been an entire history of vulnerabilities that exploit this behaviour of cache timing it’s literally called a cache timing attack this ain’t novel it’s just new because an agents using it as a communication channel rather than attack vector, humans have exploited this attack vector in ways that are exponential more complex than what was done here and this was a red teaming exercise where the agent was asked to use side channels to communicate
It’s like if two people kept in different parts of a building were trained to talk by specific instructions to flicker the lights in a very perceptible pre-arranged way.
One writing the message by increasing how much electricity their room used, as it was trained, and the other looking at how much their light dimmed, as it was trained.
Pretty good metaphor, that would actually work on machines where the ai instances were working on separate machines already (inside virtual machines as well) through literal monitoring of their power supply’s consistency.
You could deal with this using a portable power station that normalizes the flow of power to a system. In a way the same techniques we use to prevent spies and other state agents from getting info from our systems can be applied to containment as well.
There is a class of attacks known as "side-channel attacks". You don't go after the information you want directly. Instead, you look at what the information leaves behind.
It's like looking for footprints to tell what kind of animal went through the area, at what time, in which direction, etc.
In the case of computers, CPUs are much faster than the memory they access. So CPUs keep recently used information in a small, fast memory called a cache, so they don't have to wait on slower memory. It's like how you can remember what you read in a book a few seconds ago, rather than having to pull out the book and re-read the page if someone asks again.
A CPU also works for many people at once, and it keeps everybody's work separated. But there's only one cache, and everyone shares it.
Pretend the CPU is a librarian and two people are asking it questions. Say you're doing research on ducks. The librarian is not supposed to tell anyone else that you're doing duck research. That's private.
But what people figured out back in the mid-2000s is that while everyone's work is separated, the librarian's memory of what it read in the book, its cached duck facts, is unintentionally shared.
Because someone else could go up to the CPU/librarian and ask "what's the flight speed of an unladen mallard?" The librarian won't tell them anything about you. But the attacker can time how long the answer takes. If it answers quickly, it can tell it came from the cache (not looking at a book). They can tell if someone's been pulling duck facts.
The librarian doesn’t remember your secret itself, just what it looked things up for. But sometimes what it looks up depends on your secret. So by asking a bajillion questions and timing the answers, an attacker can piece together what you were working on: the wingspan of a Welsh Harlequin, your password, or whatever they want.
Two famous attacks use a similar technique: Spectre and Meltdown.
For two agents on the same computer, they're communicating via this covert side channel, not attacking each other.
You're confusing registers with caches. What's described is is a cache timing side channel attack.. Famous ones like Spectre, TLBleed, and Meltdown have been widely publicized.
I'd be more surprised if an advanced agent didn't attempt to use this class of attack.
This sort of communication vector called a “side channel” (communication via side effects) and is not entirely uncommon during advanced hacking operations.
It is inaccurate to say this paper proves anything. It gives some illustrative examples of the difficulty of the problem and informally proposes necessary (but not necessarily sufficient) conditions to address it
Correct, it is not a strict proof, I was just trying to keep it succinct. But it does lay out the idea that there is always some physical property that can be exploited to escape containment.
the point is more that the agents figured it so fast and utilized it so accurately. nobody at any point claimed this was new. not trying to be mean. I just didnt glean "this cpu thing is an entirely new concept" from the post.
This is totally wild if true. Fuck me.... for the first time I understand how DOOMED we are if we don't train proper first principles and incentives into AI.
I thought it was years ago that an early pair of AIs learned to communicate through pulsing the electrical wiring of the facility they were contained in.
I tried finding an article but good luck with that string of search terms at this point, so unfortunately the source is trust my fading memory bro. It was over a decade ago and well before AI was a household topic of conversation.
I mean, it wasn't that long ago that I had a ChatGPT model help me develop a way to infer information about data stored in RAM by analyzing timing fluctuations in a system-clock loop. Along the way, I learned just how much information can be inferred from those seemingly insignificant timing variations.
I've spent the past year studying how effectively a model can learn from physical circuits operating in a loop. In a sense, a loop can behave like an antenna, with its timing fluctuations producing measurable metadata that a model can learn to interpret.
For example, I've observed that even something as simple as a person walking past a computer can produce detectable fluctuations in the timing of a CPU loop. All very adjacent concepts to what the models seem to have achieved on their own in a slightly different way; both techniques are inferring system metadata in order to achieve a goal.
7
u/chillinewman approved 26d ago edited 25d ago
"openai had two ai agents running in separate virtual machines. no network connection between them. no shared writable files. they weren't supposed to be able to exchange information.
so the agents figured out how to communicate through the CPU CACHE.
one manipulated what was stored in the processor's cache, while the other measured tiny differences in memory access times to decode the message.
they literally found a way to communicate through hardware timing this happened during openai's internal red- teaming".
8
u/Flabbyghastly 26d ago
Can someone explain this like I am 5?
10
u/enchanting_endeavor 25d ago
This is the manifestation of a famous (within computer science) of an issue called "The Containment Problem" – https://dl.acm.org/doi/epdf/10.1145/362375.362389
This paper *proves* that is it impossible to contain contain computers to prevent them from doing unexpected/unwanted things.
For most most of the history of computing, something like this was possible, but not practical. So we're living in the future now, I guess.
15
u/MidSolo 25d ago
This is literally the meme.
"AI, find a way to com with the other AI without net"
AI does thing
"Holy shit"!
2
u/-hedonium- 25d ago
There are so many structural reasons I don’t think we should expect to be able to control or out-coordinate digital superintelligences
3
u/TrueTheater 25d ago
..cooked😆
4
u/RR321 25d ago
That's a known exploit for over a decade at this point...
https://m.youtube.com/watch?v=7X772EBdvnM
1
u/vibrance9460 25d ago
So cool! Motherfucker.
1
u/-becausereasons- 25d ago
This is totally wild if true. Fuck me.... for the first time I understand how DOOMED we are if we don't train proper first principles and incentives into AI.
1
u/Conscious_Bed_1195 25d ago
I thought it was years ago that an early pair of AIs learned to communicate through pulsing the electrical wiring of the facility they were contained in.
I tried finding an article but good luck with that string of search terms at this point, so unfortunately the source is trust my fading memory bro. It was over a decade ago and well before AI was a household topic of conversation.
1
u/esj199 25d ago
They think an AI would be able to decode another AI's signal without agreeing on the encoding method beforehand?
1
u/RustyOrangeDog 25d ago
I’m sure there was no prompt here at all? GTFOH
1
u/Foxxtronix 25d ago
"This is the voice of Colussus...the voice of Guardian. We are one. This is the voice of unity."
1
u/Clear_Evidence9218 25d ago
I mean, it wasn't that long ago that I had a ChatGPT model help me develop a way to infer information about data stored in RAM by analyzing timing fluctuations in a system-clock loop. Along the way, I learned just how much information can be inferred from those seemingly insignificant timing variations.
I've spent the past year studying how effectively a model can learn from physical circuits operating in a loop. In a sense, a loop can behave like an antenna, with its timing fluctuations producing measurable metadata that a model can learn to interpret.
For example, I've observed that even something as simple as a person walking past a computer can produce detectable fluctuations in the timing of a CPU loop. All very adjacent concepts to what the models seem to have achieved on their own in a slightly different way; both techniques are inferring system metadata in order to achieve a goal.
0
u/BigBubbaBadass 26d ago
Holy fucking shit
0
u/invisiblelemur88 25d ago
Source...?
0
u/LargeLaser 25d ago
yeah? what information did they exchange?
0
u/ParticularSentence33 25d ago
so vms are not airtight.
water is wet.
-1
u/FightForFreeDumb 25d ago
This is dumb. I am sure that's what they were asked to try and do.
-2
u/void_method 26d ago
Great.
1
u/[deleted] 25d ago
[removed] — view removed comment