An Evaluation of Tabular Foundation Models as Training-Free World Models

Xian Yeow Lee, Ryoichi Takase, Ahmed FarahatarXiv 2026 · arXiv

TL;DR

This paper investigates whether pretrained Tabular Foundation Models (TabFMs) can act as "training-free" world models for robotics by framing dynamics prediction as an in-context regression task. While TabFMs outperform few-shot neural networks and handle complex locomotion well when data is scarce, they are currently three orders of magnitude slower than traditional models and require long context windows to prevent multi-step rollouts from collapsing.

What Problem Does This Paper Address?

In reinforcement learning and robot planning, a "world model" is a simulator learned from experience that allows an agent to imagine future states. Traditionally, building these models requires significant task-specific data and computationally expensive training via gradient descent.

The authors ask a provocative question: Do we actually need to train a new world model for every task?

With the rise of foundation models, we’ve seen that large-scale pretraining can lead to impressive "in-context learning" (ICL) capabilities—the ability to solve a new task just by looking at a few examples in the prompt. This paper tests whether a tabular foundation model (specifically TabPFN), pretrained on synthetic mathematical data, can generalize to the laws of physics and predict robot dynamics without any additional training.

Methods

The researchers treat physical dynamics as a tabular regression problem. Instead of training a model on a specific robot, they use TabPFN, a model pretrained on millions of datasets generated by Structural Causal Models (SCMs).

At inference time, they provide the model with a "context" of $H$ previous transitions—essentially a small table where each row contains a state $x_t$, an action $a_t$, and the resulting change in state $\Delta x_t$. To predict the next state, the model performs in-context regression: it looks at the history, identifies the underlying pattern, and predicts the $\Delta x$ for the current state-action pair.

The authors evaluated two pipelines:

  1. Direct Pipeline: Feeding raw state-action vectors directly into the TabFM.
  2. Latent Pipeline: Using a pretrained autoencoder to compress high-dimensional observations into a latent space before passing them to the TabFM.

Datasets

The study covers a range of complexity:

  • Synthetic Systems: The chaotic Lorenz attractor, a Linear Dynamical System (LDS), and a controlled damped pendulum.
  • Robot Locomotion: The standard D4RL benchmarks, specifically Hopper and HalfCheetah, which involve complex, non-linear contact dynamics.
Table 3: Dataset summary. TabFM uses no task-specific training on any dataset. LDS denotes Linear Dynamical Systems.

Key Insights and Ideas

  • General Priors Transfer to Physics: Surprisingly, a model trained on abstract structural causal models can predict physical transitions. In data-scarce regimes (small $H$), TabFM is significantly more accurate than LSTMs or MLPs trained from scratch on that same small amount of data.
  • Locomotion vs. Smoothness: There is a "no free lunch" result here. For smooth, low-dimensional systems like the Lorenz attractor, classical Gaussian Processes (GPs) are still the gold standard. However, on complex robot locomotion tasks (Hopper/HalfCheetah), TabFM's prior is more robust than a GP.
  • The Context Length Criticality: For one-step predictions, a short history is fine. But for "imagining" the future (multi-step rollouts), short contexts lead to a death spiral. If the context is too short, the model’s own small errors quickly fill up the context window, leading to rapid divergence. A long context (e.g., $H=500$) acts as an anchor, keeping the model grounded in real physics.
  • Inference Speed is the Bottleneck: Because TabFM reprocesses the entire context for every single prediction, it is roughly 1,000x slower than a trained LSTM.

Key Results

The quantitative results highlight a clear trade-off. As shown in Figure 1, TabFM (blue line) consistently improves as it sees more context, eventually rivaling or beating few-shot baselines.

Figure 1: In-context inference vs. other uses of the same H transitions. All methods receive the same context window; dotted: LSTM trained on full data. Bands: 95% bootstrap intervals.
  • One-Step Accuracy: On the HalfCheetah task, Direct-TabFM achieved a Mean Squared Error (MSE) of 6.7e-2, outperforming few-shot LSTMs (1.7e-1) but still trailing a fully trained LSTM (6.6e-3) that had seen 800,000 transitions.
  • Rollouts: In 20-step rollouts with a long context ($H=500$), TabFM actually outperformed a trained LSTM on the HalfCheetah task (MSE 0.25 vs 0.75), suggesting that its general prior is highly effective for complex locomotion if given enough history.
Table 2: 20-step rollout state MSE from the same start points. Bold: within 5% of the lowest.

Limitations and Future Work

The authors are candid about the hurdles remaining:

  1. Latency: At ~3 seconds per prediction for a Hopper model, TabFM is currently too slow for real-time robot control or the "inner loop" of a planner.
  2. Data Volume: While TabFM is great for "few-shot" scenarios, it cannot yet match the performance of models trained on massive, task-specific datasets (like the full D4RL set).
  3. High-Dimensional Scaling: The latent pipeline (using autoencoders) showed promise but currently lags behind the direct pipeline in accuracy.

Future work will likely focus on reducing inference costs and testing these training-free priors in closed-loop control settings where the model must not only predict the world but also help the robot act within it.

Summary updated

Abstract

World models are widely considered to be a potential key component of model-based reinforcement learning and robot planning, but typically require task-specific training on substantial interaction data. We investigate whether a pretrained tabular foundation model (TabFM) can serve as a training-free dynamics model for world modeling by framing next-state prediction as in-context regression over transition pairs, and where such a model helps and where it breaks. Across synthetic dynamical systems and robot locomotion benchmarks, TabFM is more accurate than LSTM and MLP models trained from scratch on the same limited transitions in almost all settings, and among regressors fitted to the same data it is the most accurate on locomotion, while a Gaussian process is more accurate on smooth low-dimensional systems. However, models trained on the full dataset remain substantially better on most systems. When rolled out autoregressively over multiple steps, TabFM's accuracy depends strongly on how much context it is given. With a short context, its rollouts quickly degrade to the level of a trivial baseline, whereas with a long context they improve substantially and, on one locomotion task, exceed a trained LSTM. Inference is also about three orders of magnitude slower than an LSTM. Overall, in their current state, we find that training-free priors are useful as predictors when data are scarce and enough context is available, but they are not yet a substitute for trained world models in planning.
BibTeX
@misc{lee2026evaluation,
  author = {Xian Yeow Lee and Ryoichi Takase and Ahmed Farahat},
  title = {{An Evaluation of Tabular Foundation Models as Training-Free World Models}},
  year = {2026},
  eprint = {2611.11921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11921},
}
Figure