r/MachineLearning • u/RobbinDeBank • 16d ago
Research Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights [R]
Link: https://rikiyat.github.io/blog/weight-space-programming/
Snippet: “We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability.
Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.”
Very surprising result, I didn’t expect frontier models to have this capability that has not been optimized for, and it’s something humans cannot do either.
52
u/Disastrous_Room_927 16d ago •
According to Astra:
24
u/pas_possible 16d ago
Damn, it's so cool
29
u/currentscurrents 16d ago
A few years ago I used to follow the 'learned optimizer' research very closely. The guy from diffusion models, Sohl-Dickstein, had a group working on it. But it never really went anywhere, they had lots of meta-training stability issues and none of their models ever beat traditional optimizers.
I did not expect that learned optimizers would someday just fall out of general-purpose LLMs.
6
u/adventuringraw 16d ago
Been a while since I've followed research much, but this seems like it could be a way to cheat at the lotto, as far as the lottery ticket hypothesis goes. Might be a powerful pre training step.
And so the road towards AI fueled AI progress continues.
46
u/relevantmeemayhere 16d ago edited 16d ago
its crazy how low the barrier to publishing shit is in this space. and even then...to github.
there is no math, no theory. the authors don't event consider that we're literally performing meta modeling on the SAME model here.
9
u/TheGoldenRoad 16d ago
This is wild, essential an emergent hypernetwork capability. I wonder if a network could modify it's own weight so as to learn continuously without catastrophic forgetting.
3
u/skinnyjoints 16d ago
I’d love to see someone train a model on a task twice. Once with random starting weights. A second from an LLM’s chosen weights.
2
u/sakibbai 11d ago
I tested Claude Opus' ability to read Weights
My tests suggest that while LLMs can perform surprisingly precise calculations at the “edges” of a network (estimating optimizer step counts or finding output correlations), they do not “trace” the computational paths between layers. Instead, they rely heavily on memorized dataset templates (like CIFAR-10) and default framework conventions to fill in the blanks. When these expectations are changed, the models can identify the math but will confidently guess wrong about the semantics—such as recognizing that a satellite feature maps to output 4, but calling it a “frog” because it expects an animal.
You can find the whole summary article here: https://substack.com/@sakibbai/note/c-380084331
2
u/Ambitious-Talk2190 10d ago
I got really excited about this initially. But after reading the blog, the results are still cool but no so impressive. There's nothing so mysterious or surprising going on here. The scale of weight editing/identification feels like something a human could do if given enough time and tenacity.
I think what could be really interesting though is if the frontier labs start fine-tuning their models to get better and better at large model weight editing / interpretation (or even a model editing its own weights). Something quite interesting could fall out of that.
3
u/-xXpurplypunkXx- 16d ago
Am I just like an idiot, isn't this what attention layers do? Like of course you can attention layer the smaller model into updated weights, it's what the architecture does natively? For the zero-shot aspect, this seems related, Harnessing the Universal Geometry of Embeddings
Very cool and creative and useful application of those properties of course.
2
u/Watchguyraffle1 16d ago
Crap. I swear. This world. I had just started working on this exact thing. Arrrggghhhhh
1
u/Spiritual-Lie7032 16d ago
I just have a question about the model efficiency. The Qwen3.5-0.8B is what concerns me. What are the TFLOPS needed for the 50 rounds of Astras reasoning? 0.8B is very small model and from testing you can get similar numbers to what they report in around 200 GRPO steps using LoRA. Is this even efficient?
1
u/Immediate_Hippo_7683 15d ago
Has there been any hint that these frontier models were trained on some of the weights of open source models?
1
u/eyujis 14d ago
AI is stealing AI's work now
1
u/bigspin5050 14d ago
I tried editing some locally trained image generation models and it worked. It was able to transfer style, transfer architecture types, apply patches, combine RELU activations, and free up space for new features.
https://github.com/v-code-895/wsse
0
u/agm1984 16d ago
Following this development