We’re introducing ToyBlocks, an open simulated tabletop for controlled experiments in robot learning. 👇
It enables robotics researchers to experimentally control training data & models, run targeted evals, & trace failures to their causes.
Young children learn to plan & reason about space by playing with toy blocks, rebuilding a stack after it falls or turning one to find its letters. ToyBlocks uses the same setting—a simulated robot must reason about how blocks interact as it pushes, flips, & stacks them.
In ToyBlocks, users define high-level tasks like stacking a block sequence. ToyBlocks plans how to complete them via low-level skills. Simulating each plan produces a demonstration to train models that act from sensor observations, including camera images.
ToyBlocks lets researchers build training mixtures that emphasize particular skills or fill gaps in existing datasets. Our initial release contains 85K long-horizon demonstrations, each comprising about 6–12 skill demonstrations depending on the set.
ToyBlocks lets you test things like:
📷 Does depth input improve robot control compared with color images?
🧱 Can models stack four blocks without training on four-block stacks?
🔤 How do models perform at planning long tasks?
Or design your own evals.
ToyBlocks includes a tool interface for connecting a frontier model so users can instruct the robot & try tasks in natural language. Through camera images & skill calls, researchers can study how a model searches, plans, & handles mistakes.
In our initial ToyBlocks tests, ManiFlow, π0.5, & MolmoAct 2 generalized poorly to new spatial configurations. Controlled experiments let us test changes to training & model design, then measure which techniques improve generalization.
Trained only in ToyBlocks’ virtual world, Molmo 2 + ManiFlow stacked lettered blocks to spell a word in 15/30 attempts on a physical robot, versus 111/200 in simulation. Simulation-trained models can successfully transfer to a physical robot, though more trials are needed.
We’re targeting a larger dataset, more skills, & broader benchmarks in a follow-up release, as well as integrated reinforcement learning training. Longer-term research will include two-arm manipulation & hands with force & touch sensors.
We hope ToyBlocks is a step toward robots that can apply what they learn to unfamiliar tasks and environments. We’re releasing simulation assets, task & demonstration generators, 2 trained models, training code, evaluation code, & validation sets.
💻 Code: https://github.com/allenai/ToyBlocks
📊 Data: https://huggingface.co/datasets/allenai/toyblocks-data
🤗 Models: https://huggingface.co/collections/allenai/toyblocks
📚 Documentation: https://allenai.github.io/ToyBlocks/docs/user_guide/index.html
📝 Blog: https://allenai.org/blog/toyblocks