Towards Reliable Scaling Laws for Tabular Foundation Models

Johannes Hog, Neeratyoy Mallik, Steven Adriaensen, Jan Hendrik Metzen, Tom Viering, Danny Stoll, Frank HutterarXiv 2026 · arXiv

TL;DR

Scaling laws for Tabular Foundation Models (TFMs) aren't as simple as "bigger model, more data." This paper reveals that reliable scaling requires expanding the complexity of synthetic training data (more rows and features) alongside compute, and warns that using too few evaluation datasets leads to dangerously over-optimistic performance predictions.

What Problem Does This Paper Address?

Tabular Foundation Models (TFMs) like TabPFN have revolutionized tabular machine learning by performing "in-context" prediction—learning to classify new data points just by looking at a few examples, without needing task-specific training. These models are typically pretrained on virtually unlimited synthetic data generated from mathematical priors.

While scaling laws (like those for LLMs) help researchers predict how performance improves with more compute, TFMs present a unique challenge: they are trained on synthetic data but must perform on real-world data. Previous studies showed that scaling laws could be fit to TFMs, but this paper asks a harder question: Are these scaling laws reliable? Do they actually predict performance at much larger scales, and do they hold up when moving from synthetic benchmarks to messy, real-world datasets?

Methods

The researchers conducted a massive systematic study using nanoTabPFN, an open-source, simplified version of the TabPFNv2 architecture. Their methodology involved:

  • 6,000+ Training Runs: They explored a search space of eight hyperparameters, including architectural choices (layers, embedding size), training settings (learning rate, batch size), and—crucially—synthetic prior configurations.
  • Pareto Frontier Analysis: They identified the "best" model configurations for every compute budget (measured in FLOPs).
  • Extrapolation Testing: They fit scaling laws on lower-compute runs and tested their ability to predict performance at a 10x higher compute budget ($10^{17}$ FLOPs).
  • Selection Bias Quantification: They measured how the number of evaluation datasets affects the reliability of the scaling law, specifically looking for "over-fitting" to the evaluation set.

Datasets

  • Pretraining: Synthetic datasets sampled from the TabICLv1 MLP prior, which generates tabular data using random neural networks.
  • Evaluation (Synthetic): Two held-out sets of synthetic data (small and medium-sized tables) to check if the model learned the training distribution.
  • Evaluation (Real-world): The TabArena benchmark, a collection of diverse real-world tabular datasets. The authors restricted this to datasets with fewer than 500 features and 10,000 samples to match the model's design constraints.

Key Insights and Ideas

  • The "Prior" is a Scaling Axis: Unlike LLMs, where "more data" usually means more tokens, TFMs benefit from "wider" and "deeper" data. The authors found that compute-optimal scaling requires increasing the maximum number of rows and features in the synthetic training sets as the model grows.
  • Synthetic vs. Real Disconnect: Scaling trends on synthetic data don't perfectly mirror real-world performance. Synthetic evaluation tends to favor scaling the number of model parameters, while real-world benchmarks benefit more from increasing the total number of synthetic data cells seen during training.
  • The Selection Bias Trap: If you select your "optimal" model configuration based on a small handful of real-world datasets, your scaling law will be "over-optimistic." The gap between predicted and actual performance on new data only closes when you use a sufficiently large evaluation suite (at least 20 datasets for metrics like ROC AUC).
Figure 4: This graph shows how selection bias decreases as you add more evaluation datasets. If you only use 1-5 datasets, the "gap" (error) is massive compared to the actual noise floor of the scaling law.

Key Results

  • Reliable Extrapolation: The authors successfully fit power-law scaling laws ($R^2 \ge 0.986$) that accurately predicted performance at unseen compute scales. For example, the Mean Absolute Error (MAE) for ROC AUC was only 0.004 when extrapolating to $10^{17}$ FLOPs.
  • Metric Matters: Different metrics prescribe different scaling "recipes." Synthetic loss and real-world Log-Likelihood (NLL) agree closely on how to scale, but the bounded ROC AUC metric prescribes a faster growth in model size/data with compute.
Figure 1: Pareto Trends for Different Metrics. We observe typical compute-optimal performance scaling for both synthetic validation loss and aggregated metrics on real datasets. With more high-compute observations, scaling fits on any performance metric can extrapolate reliably. However, the relative scaling trends of data and parameters depend heavily on the choice of metric. Solid lines: in-sample fit; Dashed: extrapolation.

Limitations and Future Work

  • Architecture Specificity: The study focuses on TabPFNv2-style models. While these are state-of-the-art, it's unclear if these specific scaling coefficients apply to other TFM architectures like TabICL or LimiX.
  • Prior Limitations: The models were trained on a single type of synthetic prior (MLP-based). Different priors (e.g., forest-based or causal) might exhibit different scaling behaviors.
  • Future Directions: The authors suggest that future scaling laws should explicitly include the "prior support" (the range of rows/features in training) as a variable in the functional form of the law, rather than just treating it as a fixed hyperparameter.

Summary updated

Abstract

Scaling laws are widely used to predict foundation model performance and guide compute allocation. However, tabular foundation models (TFMs) differ from conventional foundation models in that they are pretrained on virtually unlimited synthetic data from configurable distributions, while ultimately being evaluated on real-world datasets. We investigate whether scaling laws for TabPFNv2-style models reliably extrapolate to larger compute budgets and generalize across evaluation settings. Our analysis reveals differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. In particular, selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates. We further demonstrate that compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. Increasing the maximum numbers of rows and features sampled during pretraining improves performance and produces more predictable scaling trends. These insights guide the training of more efficient TabPFNv2-like models, using the open-source training pipeline of nanoTabPFN.
BibTeX
@misc{hog2026towards,
  author = {Johannes Hog and Neeratyoy Mallik and Steven Adriaensen and Jan Hendrik Metzen and Tom Viering and Danny Stoll and Frank Hutter},
  title = {{Towards Reliable Scaling Laws for Tabular Foundation Models}},
  year = {2026},
  eprint = {2611.24214},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24214},
}
Figure