Abstract
The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.
Introduction
In the current era of deep learning, the "bigger is better" philosophy has driven the development of massive models trained on gargantuan datasets. However, the computational cost of training these systems is often prohibitively expensive. A central challenge for researchers is predicting whether a planned increase in model parameters or training data will yield a significant performance gain before committing substantial resources. This work by Rosenfeld et al. (ICLR 2020) addresses this by proposing a unified mathematical framework to predict generalization error across scales.
The Constructive Prediction Model
The authors propose that the generalization error $\epsilon$ can be modeled as a joint function of model size $m$ and dataset size $n$. By observing that error typically follows a power law when one variable is held constant, they derived a combined functional form:
$$\epsilon(m, n) \approx an^{-\alpha} + bm^{-\beta} + c_\infty$$
In this equation, $n$ represents the data size, $m$ the model size, and $c_\infty$ the irreducible Bayes error. To account for the "transition zones"—where a model is either too small to learn effectively (the "random guess" plateau) or large enough to hit diminishing returns—the researchers introduced a complex envelope function. This allows the model to smoothly transition from a flat error plateau into the steep power-law descent characteristic of effective scaling.
The Joint Error Landscape
A key insight of this research is that model size and data size cannot be optimized in isolation. A small model will eventually bottleneck, failing to improve regardless of how much additional data is provided. Conversely, an oversized model is inefficient if the dataset is too small to support its capacity. This relationship creates a distinct error surface.
As shown in the 3D visualization above, the error landscape forms a predictable surface. The "jointness" of the variables means that for any given compute budget, there is an optimal balance between model complexity and dataset volume. The formula captures this bottleneck relationship, anchoring predictions at the small-scale end to ensure stability when extrapolating to larger scales.
Empirical Validation
The researchers validated their formula across a diverse array of tasks, including ImageNet classification and WikiText-103 language modeling. They utilized "strict scaling" policies—such as uniformly doubling the width of every layer—to ensure that the predictions were constructive and tied to specific, reproducible architectures.
The empirical results demonstrated a remarkably high-quality fit. For ImageNet and WikiText-103, the mean relative error of the fitted formula was under 2%. The power-law behavior is clearly visible in the log-log plots, where the dotted lines represent the predicted functional form fitting the observed data points across various fractions of data and model parameters.
Extrapolation and Practical Impact
The most significant application of this work is its predictive power. The authors demonstrated that by training on only 1/16th of the parameters and 1/8th of the data, they could predict the performance of the full-scale model with less than 5% relative error. This capability allows for "pilot studies" that can justify the massive compute budgets required for state-of-the-art AI development.
Furthermore, the universality of the functional form—working across CNNs in vision and Transformers/LSTMs in language—suggests it captures a fundamental property of neural network learning dynamics.
Conclusion
By providing a robust map of the error landscape, this research moves deep learning toward a more predictable engineering discipline. While limitations remain—such as the assumption of fixed hyperparameters and the inability to model the "U-shaped" overfitting curve seen in some regimes—the proposed functional form provides a powerful tool for navigating the complexities of model and data scaling.
Summary updated