On Data Scaling in Tabular ML: More Data, Better Generalization?

Ivan Rubachev, Nikita Lazarev, Artem BabenkoarXiv 2026 · arXiv

In the world of Large Language Models, "scaling laws" are gospel: more data and more parameters almost inevitably lead to better performance. But does this hold true for the messy, heterogeneous world of tabular data?

A new paper from Yandex Research and HSE University, "On Data Scaling in Tabular ML: More Data, Better Generalization?", takes a deep dive into this question. By assembling a massive benchmark of eight real-world datasets with up to 10 million examples, the authors explore how Gradient Boosted Decision Trees (GBDTs), MLPs, and the emerging class of Tabular Foundation Models behave as they scale—especially when faced with the "distribution shifts" (changes in data over time) that plague real-world applications.

TL;DR

While more data generally improves tabular model performance, scaling introduces a treacherous "disagreement" between in-distribution and out-of-distribution (OOD) generalization. At large scales (>1M samples), optimizing a model for the best validation score on historical data can actually lead to worse performance on shifted future data—a phenomenon the authors link to the "accuracy on the wrong line" problem.

What Problem Does This Paper Address?

Most tabular ML research focuses on datasets with fewer than 1 million rows. However, industrial applications like finance, advertising, and logistics often deal with tens of millions of rows. Furthermore, these real-world tasks are rarely "stationary"; the data distribution shifts over time (e.g., consumer behavior changes, or market conditions evolve).

We don't yet fully understand if the scaling laws observed in CV or NLP apply here. Specifically:

  1. Do GBDTs and Deep Learning models benefit equally from massive data?
  2. Do "Tabular Foundation Models" (which use in-context learning) scale effectively to large contexts?
  3. Does more training data help models handle distribution shifts, or does it just make them overfit to the past?

Methods

The authors conduct a systematic study by doubling training set sizes from 10,000 to over 5 million (capping at 10M where possible). They evaluate three categories of models:

  • Parametric Models: Traditional GBDTs (XGBoost, LightGBM) and modern MLPs (including variants using the Muon optimizer and TabPack ensembles).
  • Tabular Foundation Models (PFNs): Models like TabPFN (v3, v3.5) and Kumo, which perform "training" by consuming examples in their context window.
  • Model Selection Sweeps: They ran massive sweeps (over 95,000 MLP runs) to see how model capacity (parameter count) interacts with data scale.

Datasets

The study utilizes eight large-scale datasets. Four are from the existing TabReD benchmark, and four are newly prepared or adapted by the authors to provide a cleaner signal for scaling:

  • TabReD subset: Cooking Time, Delivery ETA, Maps Routing, Weather.
  • New/Adapted:
    • Jane Street: Financial forecasting (market response prediction).
  • Aeolus Flight Delay: Predicting flight delays using 2016–2024 data.
  • Bank Fraud: Identifying fraudulent transactions.
  • Istella: A large-scale search relevance (learning-to-rank) dataset.

Key Insights and Ideas

  • Universal Scaling for Parametric Models: GBDTs and MLPs show remarkably similar scaling trends. Algorithmic improvements (like better optimizers or embeddings) tend to shift the performance curve down (improving it) without changing the "slope" of the scaling law.
  • Foundation Models are Catching Up: Newer foundation models (TabPFN v3.5, Kumo) show much more consistent gains from larger contexts than their predecessors. Some even continue to improve with up to 1 million context examples.
  • The Capacity Disagreement: This is the most striking finding. As data scales, the "best" model size depends on your evaluation goal.
  • For In-Distribution (ID) data, bigger is better: larger models continue to improve as you add data.
  • For Out-of-Distribution (OOD) data, capacity saturates quickly. Increasing model parameters beyond a moderate count (~100k) often provides zero benefit for shifted test sets, even if you have 10M rows of training data.
Figure 3: MLP width sweeps. Note how ID scores (bottom) keep improving with more parameters, while OOD scores (top) flatten out or even degrade for the largest models.

Key Results

  • Parametric Scaling: On most tasks, performance improves steadily as data grows to 10M samples. Figure 1 illustrates these consistent downward trends (lower RMSE/negative AP is better).
Figure 1: Performance of parametric models across training-set sizes. Improvements to architectures remain beneficial even at the 10M scale.
  • Foundation Model Progress: Recent models like TabPFN v3.5 and Kumo show significantly better scaling than TabICL v2 or earlier TabPFN versions.
Figure 2: Scaling of foundation models. Newer versions (v3.5) benefit much more from larger in-context training sets.
  • The Validation Trap: The authors demonstrate that using an "In-Distribution" validation set for early stopping or hyperparameter tuning at large scales can lead to a decrease in OOD performance. Essentially, the model becomes "too good" at fitting the specific quirks of the training distribution, which don't hold in the shifted test set.

Limitations and Future Work

  • Foundation Model Limits: The study caps foundation model context at 1M samples. Scaling these models to 10M+ requires new retrieval or subsampling techniques that were not explored here.
  • Validation Alignment: The paper highlights that we don't yet have a "settled" strategy for how to best use validation data that is aligned with the target distribution (e.g., should it be used for tuning, or added to the training set?).
  • Fixed Splits: To keep the signal "clean," the authors used fixed validation/test sets. In real production environments, "rolling" evaluations are more common, which might introduce even more complex temporal dynamics.

This research serves as a cautionary tale for engineers: in tabular ML, simply throwing more data and more parameters at a problem isn't a silver bullet for generalization. As your data grows, the way you select and validate your models becomes just as important as the architecture itself.

Summary updated

Abstract

Recent work in tabular machine learning has emphasized the importance of diverse and realistic evaluation settings. The niche of large-scale (1M+) data, however, remains under-explored. To study training-data scaling, we assemble eight real-world tabular datasets with up to around ten million training examples, retaining the natural distribution shifts of the tasks and using random-split evaluation as a control. We find that GBDTs and MLP-based models show broadly similar scaling trends, with gains from both additional training data and algorithmic improvements. Tabular foundation models show more varied scaling behavior, with recent versions benefiting more consistently from larger contexts even under distribution shift. However, scaling training data under distribution shift introduces new challenges: in-distribution and out-of-distribution performance mostly improve together at smaller scales but can diverge at larger scales. We examine this through MLP capacity sweeps and comparisons of the validation distributions used for tuning and early stopping in MLPs and GBDTs. We also compare common practices of using validation data, contrasting its use for model selection in MLP pipelines with using it for in-context fitting in tabular foundation model pipelines. Under stronger shifts, MLPs selected using validation representative of the evaluation conditions generally maintain or improve performance as training data grows. In contrast, including such examples in context improves foundation-model performance, but does not prevent deterioration with increasing training size in our experiments.
BibTeX
@misc{rubachev2026data,
  author = {Ivan Rubachev and Nikita Lazarev and Artem Babenko},
  title = {{On Data Scaling in Tabular ML: More Data, Better Generalization?}},
  year = {2026},
  eprint = {2610.26794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26794},
}
Figure