TL;DR
This paper investigates whether pretrained Tabular Foundation Models (TabFMs) can act as "training-free" world models for robotics by framing dynamics prediction as an in-context regression task. While TabFMs outperform few-shot neural networks and handle complex locomotion well when data is scarce, they are currently three orders of magnitude slower than traditional models and require long context windows to prevent multi-step rollouts from collapsing.
What Problem Does This Paper Address?
In reinforcement learning and robot planning, a "world model" is a simulator learned from experience that allows an agent to imagine future states. Traditionally, building these models requires significant task-specific data and computationally expensive training via gradient descent.
The authors ask a provocative question: Do we actually need to train a new world model for every task?
With the rise of foundation models, we’ve seen that large-scale pretraining can lead to impressive "in-context learning" (ICL) capabilities—the ability to solve a new task just by looking at a few examples in the prompt. This paper tests whether a tabular foundation model (specifically TabPFN), pretrained on synthetic mathematical data, can generalize to the laws of physics and predict robot dynamics without any additional training.
Methods
The researchers treat physical dynamics as a tabular regression problem. Instead of training a model on a specific robot, they use TabPFN, a model pretrained on millions of datasets generated by Structural Causal Models (SCMs).
At inference time, they provide the model with a "context" of $H$ previous transitions—essentially a small table where each row contains a state $x_t$, an action $a_t$, and the resulting change in state $\Delta x_t$. To predict the next state, the model performs in-context regression: it looks at the history, identifies the underlying pattern, and predicts the $\Delta x$ for the current state-action pair.
The authors evaluated two pipelines:
- Direct Pipeline: Feeding raw state-action vectors directly into the TabFM.
- Latent Pipeline: Using a pretrained autoencoder to compress high-dimensional observations into a latent space before passing them to the TabFM.
Datasets
The study covers a range of complexity:
- Synthetic Systems: The chaotic Lorenz attractor, a Linear Dynamical System (LDS), and a controlled damped pendulum.
- Robot Locomotion: The standard D4RL benchmarks, specifically Hopper and HalfCheetah, which involve complex, non-linear contact dynamics.
Key Insights and Ideas
- General Priors Transfer to Physics: Surprisingly, a model trained on abstract structural causal models can predict physical transitions. In data-scarce regimes (small $H$), TabFM is significantly more accurate than LSTMs or MLPs trained from scratch on that same small amount of data.
- Locomotion vs. Smoothness: There is a "no free lunch" result here. For smooth, low-dimensional systems like the Lorenz attractor, classical Gaussian Processes (GPs) are still the gold standard. However, on complex robot locomotion tasks (Hopper/HalfCheetah), TabFM's prior is more robust than a GP.
- The Context Length Criticality: For one-step predictions, a short history is fine. But for "imagining" the future (multi-step rollouts), short contexts lead to a death spiral. If the context is too short, the model’s own small errors quickly fill up the context window, leading to rapid divergence. A long context (e.g., $H=500$) acts as an anchor, keeping the model grounded in real physics.
- Inference Speed is the Bottleneck: Because TabFM reprocesses the entire context for every single prediction, it is roughly 1,000x slower than a trained LSTM.
Key Results
The quantitative results highlight a clear trade-off. As shown in Figure 1, TabFM (blue line) consistently improves as it sees more context, eventually rivaling or beating few-shot baselines.
- One-Step Accuracy: On the HalfCheetah task, Direct-TabFM achieved a Mean Squared Error (MSE) of 6.7e-2, outperforming few-shot LSTMs (1.7e-1) but still trailing a fully trained LSTM (6.6e-3) that had seen 800,000 transitions.
- Rollouts: In 20-step rollouts with a long context ($H=500$), TabFM actually outperformed a trained LSTM on the HalfCheetah task (MSE 0.25 vs 0.75), suggesting that its general prior is highly effective for complex locomotion if given enough history.
Limitations and Future Work
The authors are candid about the hurdles remaining:
- Latency: At ~3 seconds per prediction for a Hopper model, TabFM is currently too slow for real-time robot control or the "inner loop" of a planner.
- Data Volume: While TabFM is great for "few-shot" scenarios, it cannot yet match the performance of models trained on massive, task-specific datasets (like the full D4RL set).
- High-Dimensional Scaling: The latent pipeline (using autoencoders) showed promise but currently lags behind the direct pipeline in accuracy.
Future work will likely focus on reducing inference costs and testing these training-free priors in closed-loop control settings where the model must not only predict the world but also help the robot act within it.
Summary updated