In the early days of the deep learning revolution, the recipe for success seemed simple: if you want a better model, make it bigger. This intuition was codified into "scaling laws" that suggested performance improves predictably as a power law of compute, parameters, and data. However, as models have pushed into the hundreds of billions of parameters, a more nuanced reality has emerged.
Research across diverse fields—from computer vision to protein folding—now suggests that model capacity and data volume exist in a delicate co-dependency. Scaling one without the other does not just lead to inefficient training; it often leads to a hard performance ceiling known as saturation.
The Capacity Data Seesaw
The fundamental intuition behind modern scaling research is that a model’s parameter count represents its "learning capacity," while the dataset represents the "signal" it must capture. If the capacity is too small, the model cannot capture the nuances of the data (underfitting). If the capacity is too large relative to the data, the model begins to memorize noise or specific quirks of the training set rather than generalizable patterns (overfitting).
Hao Li 0015, Shamit Lal, and their colleagues demonstrated this in Efficient Scaling of Diffusion Transformers for Text-to-Image Generation. They found that scaling Diffusion Transformers (DiTs) beyond 8 billion parameters yielded diminishing returns unless the training data volume was increased proportionally. Essentially, the architectural capacity outpaced the available training signal.
This isn't a new phenomenon, but rather a persistent constraint of neural networks. As early as 2015, Xiaogang Wang 0001 and Yi Sun noted in DeepID3: Face Recognition with Very Deep Neural Networks that the high learning capacities of very deep networks could only be fully realized if the training data volume increased in tandem.
Identifying the Bottleneck Regime
To understand why a model stops improving, researchers look at which side of the "seesaw" is currently the bottleneck. Jared Kaplan and Utkarsh Sharma characterize this in A Neural Scaling Law from the Dimension of the Data Manifold. They argue that models often transition from a "parameter-limited" regime (where adding more neurons helps) to a "data-limited" regime (where the dataset is too small to support further expansion).
When a model is under-trained, it leaves performance on the table. For example, Aohan Zeng and Xiao Liu observed in GLM-130B: An Open Bilingual Pre-trained Model that even massive 130B parameter models can remain under-trained if the token volume does not align with optimal scaling ratios. Conversely, in self-supervised vision, Zhenda Xie and Yue Cao 0001 showed in On Data Scaling in Masked Image Modeling that increasing model size without enough data leads to a divergence between training and validation loss—the classic signature of overfitting.
Domain Specific Saturation
The "sweet spot" for this balance varies wildly depending on the complexity of the task. In specialized domains, the bottleneck is often the availability of high-quality, diverse data rather than raw compute.
- Vision-Language: Xiaowei Hu 0006 and Zhe Gan demonstrated in Scaling Up Vision-Language Pretraining for Image Captioning that image captioning follows predictable laws, but larger architectures only start to shine once a specific data threshold is crossed.
- Geospatial Forecasting: In Forecasting localized weather impacts on vegetation, Zhihan Gao 0001 found that large transformers hit plateaus because the training data lacked the diversity to match the model's complexity.
- Scientific Modeling: Nathan C. Frey and colleagues explored this in Neural scaling of deep chemical models, noting that scaling laws "break" when there is a mismatch between the model's capacity and the resolution of the chemical dataset.
Beyond Data Volume: The Role of Diversity
Recent work suggests that simply adding more data isn't enough; the data must be sufficiently diverse to "fill" the model's capacity. Philipe Ambrozio Dias and Aristeidis Tsaris argue in OReole-FM that billion-parameter foundation models for satellite imagery require a careful balance of geographic and temporal variety. Without this variety, the model overfits to specific regions, and the benefits of scaling the parameter count vanish.
This shift in perspective—from "bigger is better" to "balanced is better"—is forcing researchers to rethink how they allocate resources. As Ayush Jain and Andrea Montanari point out in Scaling laws for learning with real and surrogate data, we must incorporate model-size terms into our predictions to accurately forecast performance in an era where data, not just compute, is the ultimate bottleneck.