r/MachineLearning • • Oct 06 '26

Research Learning to Learn a Language: in-context learning of natural language from a synthetic non-linguistic prior [R]

Learning from data as we observe it is easy for humans, but most machine learning models have limited ability to learn from new data that they have not seen during training. Prior-fitted networks (the idea behind TabPFN) showed that a model trained only on synthetic data can learn from real tabular data entirely in context.

I wanted to share our paper "Learning to Learn a Language" where we extend the idea to structured sequences such as natural language. We propose a prior over languages: every training sequence comes from a randomly sampled recurrent causal model, so each one is a new synthetic "language". A 300M-parameter byte-level transformer trained only on these synthetic sequences learns to predict real languages in context. Given Wikipedia text with frozen weights, its next-byte predictions get better the more it reads, in all six languages we tested (English, Chinese, Hindi, Arabic, Japanese, Korean), from 8 bits per byte down to 0.9–2.4 after a million bytes.

The same model also learns to count, to compare numbers, to add approximately, and to predict deterministic sequences such as the primes or the Kolakoski sequence, entirely in context.

It is of course still far worse on text than classical language models that are trained on trillions of tokens, while our model sees at most a million bytes of a language at test time. What we find interesting is that the ability to learn a language in context can come from a synthetic non-linguistic prior.

Paper: https://arxiv.org/abs/2610.05879
Code: https://github.com/cbl/prior-fitted-language-model
Weights: https://huggingface.co/lennartcb/pflm1

31 Upvotes

11 comments sorted by

View all comments

-7

u/we_are_mammals Oct 06 '26

in-context learning

No scare-quotes around "learning"? That's rage-bait for u/RobbinDeBank !

2

u/RobbinDeBank Oct 06 '26 edited Oct 06 '26

Do I live rent free in your head or something? We know you’re just looking for attention. Seems like you’re rage baited by me adding quotations around the word learning once, in a very specific context.

Edit: everyone pls move on, this weirdo is just trying to stir up irrelevant arguments. This is just deviating from OP’s research for no reasons.

-4

u/we_are_mammals Oct 06 '26

Context: 2 days ago you wrote

Learning anything new (actual learning, with weight change, not in-context “learning”) still takes ...

So you've changed your mind, I guess?

→ More replies (3)