TL;DR
Tabular Foundation Models (TFMs) excel at interpolating physical data out of the box, frequently outperforming heavily tuned traditional machine learning baselines. However, they fail dramatically at extrapolation because their underlying Bayesian priors cannot represent noiseless deterministic mechanisms or exploit physical units and dimensional structure. Ultimately, while TFMs are powerful statistical interpolators, they do not truly "know" or simulate underlying physical laws.
What Problem Does This Paper Address?
In the physical sciences, data overwhelmingly arrives in tabular form. While Large Language Models (LLMs) have made massive strides in mathematical text and vision models dominate scientific imaging, Tabular Foundation Models (TFMs)—which use in-context learning to fill in missing table cells—have received far less scrutiny.
This paper investigates a fundamental question: Do TFMs implicitly learn physics just by learning to predict tabular data? Because TFMs are Bayesian by construction, their success depends entirely on what their pretraining prior contains. The authors test whether these priors allow TFMs to act as genuine physical models capable of generalizing to unseen regimes (extrapolation), or if they are simply highly effective curve-fitters (interpolation).
Methods
The authors evaluate four prominent open-weight TFMs: TabPFN-3, TabICLv2, TabDPT, and Real-TabPFN-2.5. They compare them against six state-of-the-art tabular baselines (including CatBoost, Gaussian Processes, and MLPs) across three tuning pipelines: library defaults, random-searched hyperparameter tuning, and tuned post-hoc ensembling.
To ensure a rigorous assessment, the methodology includes:
- A Contamination Audit: A rigorous check to determine if real-data pretrained models had previously seen the evaluation equations, measuring performance gaps between "seen" and "unseen" structurally similar datasets.
- In-Domain vs. Extrapolation Splits: Testing models both within the training data bounding box (in-domain, $k=1$) and in an external bounding shell (extrapolation, $k=2$).
- Noise and Sample Sweeps: Varying the number of training samples ($n \in \{50, 200, 2000\}$) and relative multiplicative noise ($\sigma \in \{0, 0.01, 0.1\}$) to analyze how models behave in the perfectly deterministic limit ($\sigma = 0$).
Datasets
Rather than using static, pre-existing benchmarks where data contamination is a risk, the authors dynamically regenerate fresh data from 316 physical equations across three suites:
- Feynman (120 equations): Standard physical laws spanning classical to quantum physics, derived from the Feynman Lectures on Physics.
- LSR-Transform (111 equations): Feynman equations mathematically rewritten into unusual, non-standard forms.
- LSR-Synth (85 equations): Newly composed, synthetically generated equations featuring novel combinations of terms to test true out-of-domain generalization.
Key Insights and Ideas
- The Noiseless Ceiling: In pure physics, a deterministic law implies zero aleatoric noise ($\sigma = 0$). However, the authors discover that every TFM reports a non-zero predictive uncertainty width that plateaus and refuses to shrink, even as more data samples are provided. Their priors inherently assume that some noise always exists, preventing them from capturing perfect deterministic relationships.
- Blind to Dimensions and Units: Physical variables carry dimensions (mass, length, time). According to Buckingham’s $\pi$ theorem, physical laws can be simplified into a smaller set of dimensionless groups. While traditional models like CatBoost turn this dimensional reducibility to their advantage, the strongest TFM (TabPFN-3) actually performs worse on tasks with higher dimensional reducibility.
- The Illusion of Extrapolation: TFMs are master interpolators because their synthetic pretraining covers a massive variety of causal graphs. However, because they lack structural knowledge of periodic or physical constraints, their predictions immediately decay to a flat constant baseline once pushed outside the training domain.
Key Results
- TFM Dominance in Interpolation: Out of the box, TFMs dominate. TabPFN-3 achieved the top mean rank across almost all in-domain strata, outperforming even fully tuned and ensembled Gaussian Processes and gradient-boosted trees.
- Immediate Extrapolation Failure: When tested on a noiseless damped harmonic oscillator, all four TFMs failed to continue the oscillation outside the sampled range, relaxing to a constant value within roughly half a period.
- No Data Contamination: The contamination audit confirmed that the high performance of real-data TFMs was genuine and not due to data leaks; the performance metrics on "seen" datasets fell entirely within the expected unexposed baseline bands.
Limitations and Future Work
The core limitation is a fundamental mismatch in the pretraining data. Current TFMs are pretrained either on general real-world classification/regression tables or synthetic structural causal models (SCMs) that completely lack physical constants, units, or continuous deterministic targets.
The authors suggest that to build true "Physical TFMs," future pretraining corpora must explicitly incorporate physical laws, continuous targets, and dimensional structures into the model's prior. Only then can these foundation models transition from excellent statistical interpolators into robust scientific simulators.
Summary updated