In the world of Large Language Models, "scaling laws" are gospel: more data and more parameters almost inevitably lead to better performance. But does this hold true for the messy, heterogeneous world of tabular data?
A new paper from Yandex Research and HSE University, "On Data Scaling in Tabular ML: More Data, Better Generalization?", takes a deep dive into this question. By assembling a massive benchmark of eight real-world datasets with up to 10 million examples, the authors explore how Gradient Boosted Decision Trees (GBDTs), MLPs, and the emerging class of Tabular Foundation Models behave as they scale—especially when faced with the "distribution shifts" (changes in data over time) that plague real-world applications.
TL;DR
While more data generally improves tabular model performance, scaling introduces a treacherous "disagreement" between in-distribution and out-of-distribution (OOD) generalization. At large scales (>1M samples), optimizing a model for the best validation score on historical data can actually lead to worse performance on shifted future data—a phenomenon the authors link to the "accuracy on the wrong line" problem.
What Problem Does This Paper Address?
Most tabular ML research focuses on datasets with fewer than 1 million rows. However, industrial applications like finance, advertising, and logistics often deal with tens of millions of rows. Furthermore, these real-world tasks are rarely "stationary"; the data distribution shifts over time (e.g., consumer behavior changes, or market conditions evolve).
We don't yet fully understand if the scaling laws observed in CV or NLP apply here. Specifically:
- Do GBDTs and Deep Learning models benefit equally from massive data?
- Do "Tabular Foundation Models" (which use in-context learning) scale effectively to large contexts?
- Does more training data help models handle distribution shifts, or does it just make them overfit to the past?
Methods
The authors conduct a systematic study by doubling training set sizes from 10,000 to over 5 million (capping at 10M where possible). They evaluate three categories of models:
- Parametric Models: Traditional GBDTs (XGBoost, LightGBM) and modern MLPs (including variants using the Muon optimizer and TabPack ensembles).
- Tabular Foundation Models (PFNs): Models like TabPFN (v3, v3.5) and Kumo, which perform "training" by consuming examples in their context window.
- Model Selection Sweeps: They ran massive sweeps (over 95,000 MLP runs) to see how model capacity (parameter count) interacts with data scale.
Datasets
The study utilizes eight large-scale datasets. Four are from the existing TabReD benchmark, and four are newly prepared or adapted by the authors to provide a cleaner signal for scaling:
- TabReD subset: Cooking Time, Delivery ETA, Maps Routing, Weather.
- New/Adapted:
- Jane Street: Financial forecasting (market response prediction).
- Aeolus Flight Delay: Predicting flight delays using 2016–2024 data.
- Bank Fraud: Identifying fraudulent transactions.
- Istella: A large-scale search relevance (learning-to-rank) dataset.
Key Insights and Ideas
- Universal Scaling for Parametric Models: GBDTs and MLPs show remarkably similar scaling trends. Algorithmic improvements (like better optimizers or embeddings) tend to shift the performance curve down (improving it) without changing the "slope" of the scaling law.
- Foundation Models are Catching Up: Newer foundation models (TabPFN v3.5, Kumo) show much more consistent gains from larger contexts than their predecessors. Some even continue to improve with up to 1 million context examples.
- The Capacity Disagreement: This is the most striking finding. As data scales, the "best" model size depends on your evaluation goal.
- For In-Distribution (ID) data, bigger is better: larger models continue to improve as you add data.
- For Out-of-Distribution (OOD) data, capacity saturates quickly. Increasing model parameters beyond a moderate count (~100k) often provides zero benefit for shifted test sets, even if you have 10M rows of training data.
Key Results
- Parametric Scaling: On most tasks, performance improves steadily as data grows to 10M samples. Figure 1 illustrates these consistent downward trends (lower RMSE/negative AP is better).
- Foundation Model Progress: Recent models like TabPFN v3.5 and Kumo show significantly better scaling than TabICL v2 or earlier TabPFN versions.
- The Validation Trap: The authors demonstrate that using an "In-Distribution" validation set for early stopping or hyperparameter tuning at large scales can lead to a decrease in OOD performance. Essentially, the model becomes "too good" at fitting the specific quirks of the training distribution, which don't hold in the shifted test set.
Limitations and Future Work
- Foundation Model Limits: The study caps foundation model context at 1M samples. Scaling these models to 10M+ requires new retrieval or subsampling techniques that were not explored here.
- Validation Alignment: The paper highlights that we don't yet have a "settled" strategy for how to best use validation data that is aligned with the target distribution (e.g., should it be used for tuning, or added to the training set?).
- Fixed Splits: To keep the signal "clean," the authors used fixed validation/test sets. In real production environments, "rolling" evaluations are more common, which might introduce even more complex temporal dynamics.
This research serves as a cautionary tale for engineers: in tabular ML, simply throwing more data and more parameters at a problem isn't a silver bullet for generalization. As your data grows, the way you select and validate your models becomes just as important as the architecture itself.
Summary updated