Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limit

Wassim Tenachi, Yashar Hezaveh, Laurence Perreault Levasseur, Pierre-Luc BaconarXiv 2026 · arXiv

TL;DR

Tabular Foundation Models (TFMs) excel at interpolating physical data out of the box, frequently outperforming heavily tuned traditional machine learning baselines. However, they fail dramatically at extrapolation because their underlying Bayesian priors cannot represent noiseless deterministic mechanisms or exploit physical units and dimensional structure. Ultimately, while TFMs are powerful statistical interpolators, they do not truly "know" or simulate underlying physical laws.

What Problem Does This Paper Address?

In the physical sciences, data overwhelmingly arrives in tabular form. While Large Language Models (LLMs) have made massive strides in mathematical text and vision models dominate scientific imaging, Tabular Foundation Models (TFMs)—which use in-context learning to fill in missing table cells—have received far less scrutiny.

This paper investigates a fundamental question: Do TFMs implicitly learn physics just by learning to predict tabular data? Because TFMs are Bayesian by construction, their success depends entirely on what their pretraining prior contains. The authors test whether these priors allow TFMs to act as genuine physical models capable of generalizing to unseen regimes (extrapolation), or if they are simply highly effective curve-fitters (interpolation).

Methods

The authors evaluate four prominent open-weight TFMs: TabPFN-3, TabICLv2, TabDPT, and Real-TabPFN-2.5. They compare them against six state-of-the-art tabular baselines (including CatBoost, Gaussian Processes, and MLPs) across three tuning pipelines: library defaults, random-searched hyperparameter tuning, and tuned post-hoc ensembling.

To ensure a rigorous assessment, the methodology includes:

  1. A Contamination Audit: A rigorous check to determine if real-data pretrained models had previously seen the evaluation equations, measuring performance gaps between "seen" and "unseen" structurally similar datasets.
  2. In-Domain vs. Extrapolation Splits: Testing models both within the training data bounding box (in-domain, $k=1$) and in an external bounding shell (extrapolation, $k=2$).
  3. Noise and Sample Sweeps: Varying the number of training samples ($n \in \{50, 200, 2000\}$) and relative multiplicative noise ($\sigma \in \{0, 0.01, 0.1\}$) to analyze how models behave in the perfectly deterministic limit ($\sigma = 0$).

Datasets

Rather than using static, pre-existing benchmarks where data contamination is a risk, the authors dynamically regenerate fresh data from 316 physical equations across three suites:

  • Feynman (120 equations): Standard physical laws spanning classical to quantum physics, derived from the Feynman Lectures on Physics.
  • LSR-Transform (111 equations): Feynman equations mathematically rewritten into unusual, non-standard forms.
  • LSR-Synth (85 equations): Newly composed, synthetically generated equations featuring novel combinations of terms to test true out-of-domain generalization.

Key Insights and Ideas

  • The Noiseless Ceiling: In pure physics, a deterministic law implies zero aleatoric noise ($\sigma = 0$). However, the authors discover that every TFM reports a non-zero predictive uncertainty width that plateaus and refuses to shrink, even as more data samples are provided. Their priors inherently assume that some noise always exists, preventing them from capturing perfect deterministic relationships.
  • Blind to Dimensions and Units: Physical variables carry dimensions (mass, length, time). According to Buckingham’s $\pi$ theorem, physical laws can be simplified into a smaller set of dimensionless groups. While traditional models like CatBoost turn this dimensional reducibility to their advantage, the strongest TFM (TabPFN-3) actually performs worse on tasks with higher dimensional reducibility.
  • The Illusion of Extrapolation: TFMs are master interpolators because their synthetic pretraining covers a massive variety of causal graphs. However, because they lack structural knowledge of periodic or physical constraints, their predictions immediately decay to a flat constant baseline once pushed outside the training domain.

Key Results

  • TFM Dominance in Interpolation: Out of the box, TFMs dominate. TabPFN-3 achieved the top mean rank across almost all in-domain strata, outperforming even fully tuned and ensembled Gaussian Processes and gradient-boosted trees.
  • Immediate Extrapolation Failure: When tested on a noiseless damped harmonic oscillator, all four TFMs failed to continue the oscillation outside the sampled range, relaxing to a constant value within roughly half a period.
Figure 1: TFMs interpolate physics but do not learn enough of it to extrapolate. A noiseless damped harmonic oscillator queried beyond the sampled range, against baselines. The oscillation is recovered where there is data and lost immediately outside it. Details and other models in App. A.
  • No Data Contamination: The contamination audit confirmed that the high performance of real-data TFMs was genuine and not due to data leaks; the performance metrics on "seen" datasets fell entirely within the expected unexposed baseline bands.

Limitations and Future Work

The core limitation is a fundamental mismatch in the pretraining data. Current TFMs are pretrained either on general real-world classification/regression tables or synthetic structural causal models (SCMs) that completely lack physical constants, units, or continuous deterministic targets.

The authors suggest that to build true "Physical TFMs," future pretraining corpora must explicitly incorporate physical laws, continuous targets, and dimensional structures into the model's prior. Only then can these foundation models transition from excellent statistical interpolators into robust scientific simulators.

Summary updated

Abstract

Tabular foundation models (TFMs) learn to fill in tables the way language models fill in text, and tables are arguably the format in which most physical measurement arrives. Did they learn any physics in the process? They are Bayesian by construction, so the question is what their prior contains. We probe it directly, evaluating four of them (TabPFN-3, TabICLv2, TabDPT and Real-TabPFN-2.5) against six baselines on datasets sampled from 316 physical equations, in and out of domain. TFMs dominate, out of the box and after tuning. But we show that their prior can represent neither a noiseless mechanism nor physical units, which is why they interpolate physics without yet being able to act as physical models.
BibTeX
@misc{tenachi2026do,
  author = {Wassim Tenachi and Yashar Hezaveh and Laurence Perreault Levasseur and Pierre-Luc Bacon},
  title = {{Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limit}},
  year = {2026},
  eprint = {2609.02766},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.02766},
}
Figure