Optimal Scaling of Parameters and Training Data

This direction is split into other directions: model capacity bottleneck. Its last version stays readable and citable here.

Version 1 of 1606 papers730 conceptsfrozen 2026-10-04

In the early days of the deep learning revolution, the recipe for success seemed simple: if you want a better model, make it bigger. This intuition was codified into "scaling laws" that suggested performance improves predictably as a power law of compute, parameters, and data. However, as models have pushed into the hundreds of billions of parameters, a more nuanced reality has emerged.

Research across diverse fields—from computer vision to protein folding—now suggests that model capacity and data volume exist in a delicate co-dependency. Scaling one without the other does not just lead to inefficient training; it often leads to a hard performance ceiling known as saturation.

The Capacity Data Seesaw

The fundamental intuition behind modern scaling research is that a model’s parameter count represents its "learning capacity," while the dataset represents the "signal" it must capture. If the capacity is too small, the model cannot capture the nuances of the data (underfitting). If the capacity is too large relative to the data, the model begins to memorize noise or specific quirks of the training set rather than generalizable patterns (overfitting).

Hao Li 0015, Shamit Lal, and their colleagues demonstrated this in Efficient Scaling of Diffusion Transformers for Text-to-Image Generation. They found that scaling Diffusion Transformers (DiTs) beyond 8 billion parameters yielded diminishing returns unless the training data volume was increased proportionally. Essentially, the architectural capacity outpaced the available training signal.

This isn't a new phenomenon, but rather a persistent constraint of neural networks. As early as 2015, Xiaogang Wang 0001 and Yi Sun noted in DeepID3: Face Recognition with Very Deep Neural Networks that the high learning capacities of very deep networks could only be fully realized if the training data volume increased in tandem.

Identifying the Bottleneck Regime

To understand why a model stops improving, researchers look at which side of the "seesaw" is currently the bottleneck. Jared Kaplan and Utkarsh Sharma characterize this in A Neural Scaling Law from the Dimension of the Data Manifold. They argue that models often transition from a "parameter-limited" regime (where adding more neurons helps) to a "data-limited" regime (where the dataset is too small to support further expansion).

When a model is under-trained, it leaves performance on the table. For example, Aohan Zeng and Xiao Liu observed in GLM-130B: An Open Bilingual Pre-trained Model that even massive 130B parameter models can remain under-trained if the token volume does not align with optimal scaling ratios. Conversely, in self-supervised vision, Zhenda Xie and Yue Cao 0001 showed in On Data Scaling in Masked Image Modeling that increasing model size without enough data leads to a divergence between training and validation loss—the classic signature of overfitting.

Domain Specific Saturation

The "sweet spot" for this balance varies wildly depending on the complexity of the task. In specialized domains, the bottleneck is often the availability of high-quality, diverse data rather than raw compute.

Beyond Data Volume: The Role of Diversity

Recent work suggests that simply adding more data isn't enough; the data must be sufficiently diverse to "fill" the model's capacity. Philipe Ambrozio Dias and Aristeidis Tsaris argue in OReole-FM that billion-parameter foundation models for satellite imagery require a careful balance of geographic and temporal variety. Without this variety, the model overfits to specific regions, and the benefits of scaling the parameter count vanish.

This shift in perspective—from "bigger is better" to "balanced is better"—is forcing researchers to rethink how they allocate resources. As Ayush Jain and Andrea Montanari point out in Scaling laws for learning with real and surrogate data, we must incorporate model-size terms into our predictions to accurately forecast performance in an era where data, not just compute, is the ultimate bottleneck.

Papers in this version

Showing 60 of 606 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimal_scaling_of_parameters_and_traini_ccebf168,
  title = {Optimal Scaling of Parameters and Training Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-scaling-of-parameters-and-training-data/rdr_ccebf168542c45f8955de5c06dc3037c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimal_scaling_of_parameters_and_traini_ccebf168_v1,
  title = {Optimal Scaling of Parameters and Training Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-scaling-of-parameters-and-training-data/rdr_ccebf168542c45f8955de5c06dc3037c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}