Learning Robot Behavior Priors from Non-Robotic Synthetic Data

Mateusz Odrowaz-Sypniewski, Michael Amir, Jasmine Bayrooti, Ajay Shankar, Amanda ProrokarXiv 2026 · arXiv

TL;DR

This paper demonstrates that robot models can acquire fundamental "control priors" by training exclusively on abstract, non-robotic synthetic data like Linear Dynamical Systems and Markov Decision Processes. These models achieve non-trivial zero-shot action prediction on real-world robots—ranging from quadrupeds to multicopters—often out-generalizing models trained on specific robotic datasets when faced with entirely new embodiments.

What Problem Does This Paper Address?

Training generalist robot models is currently bottlenecked by the scarcity and expense of high-quality, diverse robotic data. While "foundation models" for robotics (VLAs) leverage internet-scale text and video, they still struggle with the "physical grounding" required for precise control across different robot shapes (morphologies) and tasks.

The authors ask a provocative question: Can a model learn the "grammar" of control from pure mathematics, without ever seeing a robot? They investigate whether inexpensive, procedurally generated synthetic data—devoid of any physical embodiment—can provide a useful starting point for zero-shot transfer to real-world robotic systems.

Methods

The researchers utilize Prior-Data Fitted Networks (PFNs), a class of Transformers designed for in-context learning. Instead of being fine-tuned on a specific dataset, these models are trained to look at a "context" of examples and predict the next output in a single forward pass.

To build their "control prior," they developed three synthetic data generators based on mathematical abstractions:

  1. Markov Decision Process (MDP): Models goal-directed, sequential decision-making where actions depend on long-term value.
  2. Contextual Bandit: Isolates state-action associations by removing temporal dynamics.
  3. Linear Dynamical System (LDS): Models continuous dynamics with high temporal autocorrelation, mimicking the "smoothness" of physical motion.

Crucially, these generators do not model kinematic chains, sensors, or specific tasks. The model must infer the underlying structure of a new robot solely from the provided in-context samples.

Predictions made by the model (green arrows) vs. ground-truth (black arrows) in various environments. The model shows a clear ability to capture the direction and magnitude of motion without prior exposure to these specific systems.

Datasets

The models were evaluated on a diverse suite of held-out robotic datasets:

  • Simulated: VMAS (multi-agent RL) and RoboMimic (manipulation tasks like 'can' and 'square').
  • Real-World: DROID (7-DoF manipulator), A2D2 (autonomous driving), Aerial (multicopter flight), and GrandTour (ANYmal quadruped).

Key Insights and Ideas

  • LDS is the "Secret Sauce": Ablation studies (Figure 3) revealed that the Linear Dynamical System (LDS) prior is the most effective individual component for real-robot transfer. This suggests that the most critical "prior" for robotics is the understanding of local temporal continuity and smooth state evolution in continuous space.
  • Generalization vs. Specialization: Models trained on real robot data (the "Embodied Baseline") are superior on the specific tasks they were trained on. However, they often collapse to "worse than random" performance on new tasks. In contrast, the synthetic-only model performs consistently across all tasks because every task is equally "new" to it.
  • Context Matters: The model's accuracy increases monotonically as the context window grows (up to 1000 samples), proving it is actively learning the specific robot's behavior in-context rather than relying on a static rule.
Fig. 3: Ablation of the individual control priors on RoboMimic and real- world robotic datasets, alongside their combination and the tabular prior.

Key Results

  • Zero-Shot Success: On real-world robotic tasks with 64 possible action tokens, the synthetic-only model achieved ~17% accuracy. While this sounds low, it is 10x higher than random chance (~1.6%) and was achieved without the model ever seeing a single real robot during training.
  • Outperforming Baselines: In the VMAS simulation, the synthetic model achieved 61% accuracy on held-out tasks, significantly beating the embodied baseline which managed only 27% (Figure 4).
  • Continuous Structure: The model doesn't just guess the right discrete "token"; its errors are spatially close to the truth. The Relative Centroid Distance (RCD) metric shows the model captures the underlying continuous spatial structure of the action space.

Limitations and Future Work

  • Absolute Performance: While the zero-shot signal is "meaningful," 17% accuracy is not yet sufficient for high-stakes autonomous deployment without further refinement.
  • The Fine-Tuning Paradox: Surprisingly, the authors found that naive fine-tuning of the synthetic prior on real data was less effective than training from scratch on large real datasets. They hypothesize a conflict between the "broad generalization" objective of synthetic pretraining and the "specific optimization" of fine-tuning.
  • Scaling: The experiments used a relatively small Transformer architecture. A major next step is determining if these mathematical control priors scale effectively to massive foundation models.

These findings suggest that the path to zero-shot physical AI might depend less on collecting millions of hours of robot video and more on identifying the right mathematical abstractions of control.

Summary updated

Abstract

Real robot data, rich across tasks and morphologies, is often treated as the bottleneck for training robot learning models, including embodied foundation models that aim for zero-shot physical transfer. In this paper, we investigate the degree to which such models can acquire necessary structure using data from systems without embodiment. To this end, we test if inexpensive, procedurally generated synthetic data can yield useful priors before a model observes any real-world behavior. We study this through robot action prediction, a natural pretraining objective and a direct test of whether structure learned from arbitrary synthetic tasks transfers to embodied settings. We introduce three data generation pipelines that model mathematical abstractions of robotic control, and use them as pretraining priors for a Transformer. We demonstrate that a model trained purely on this non-robotic data attains non-trivial zero-shot prediction performance on unseen physical robot tasks via an in-context learning mechanism, and that our specialized control priors notably improve upon generic causal priors. We also compare against a compute-matched baseline trained on real robot data, and find that it outperforms our model but only on the datasets it was trained on. These findings suggest that procedurally generated, mathematical abstractions can yield useful robotic priors, offering a potential complement to expensive embodied data collection.
BibTeX
@misc{odrowazsypniewski2026learning,
  author = {Mateusz Odrowaz-Sypniewski and Michael Amir and Jasmine Bayrooti and Ajay Shankar and Amanda Prorok},
  title = {{Learning Robot Behavior Priors from Non-Robotic Synthetic Data}},
  year = {2026},
  eprint = {2610.31275},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.31275},
}
Figure