Unified Neural Network Scaling Laws and Scale-time Equivalence

Akhilan Boopathy, Ila Fiete

Introduction

In the pursuit of increasingly powerful artificial intelligence, researchers have relied on scaling laws to predict model performance based on size and data. However, existing frameworks often fail to account for the complex trade-off between model size and training duration, nor do they adequately explain the phenomenon of "double descent." This paper introduces a unified framework centered on "scale-time equivalence," suggesting that increasing a model's size is functionally equivalent to training a smaller model for a proportionally longer duration.

Theoretical Framework: Scale-Time Equivalence

The core of this research is the discovery that model scale ($p$) and training time ($t$) are intrinsically linked. Using a Random Subspace Model, the authors theoretically demonstrate that in linear settings, the learned function depends primarily on the product of parameters and training iterations ($p \times t = \text{constant}$).

To apply this to deep neural networks, the authors introduce the concept of Effective Parameter Count, proposing that a network's effective scale is proportional to the cube root of its total parameters. This leads to a unified scaling law that characterizes test error as a competition between signal acquisition and noise overfitting.

Key Insights

The "Scale is Time" insight implies that larger models learn faster. Specifically, doubling the effective size of a model roughly doubles its learning speed. This relationship provides a new lens through which to view Double Descent. The characteristic "bump" in error occurs because smaller models traverse the "danger zone"—where they have captured noise but not yet mastered complex signals—more slowly. Larger models, by virtue of their scale, move through this phase rapidly, effectively "skipping" the worst of the overfitting.

Figure 1: Proportional trade-off between model scale and training time in a linear model. The grey dashed lines show the predicted 1:1 relationship.

Empirical Validation and Extrapolation

The researchers validated these findings across standard benchmarks including MNIST, CIFAR-10, and SVHN using CNN and MLP architectures. A key practical result is the ability to perform Extrapolation: using the $p \times t$ relationship, the performance of massive models (trained for few epochs) can be accurately predicted using data from smaller models trained for many epochs. This allows for significant efficiency gains in the development of large-scale models.

Figure 3: Using scale-time equivalence to predict the performance of large models from small ones and vice versa.

Furthermore, the study confirms that larger models are more data-efficient. They reach the interpolation threshold—the point at which they can fit the training data—with significantly less data than their smaller counterparts. However, in the presence of label noise, this efficiency can lead to a "U-shaped" trap where larger models overfit to noise more quickly if training is not carefully regulated.

Discussion and Limitations

While the scale-time equivalence is robust under Stochastic Gradient Descent (SGD), the relationship becomes more complex with adaptive optimizers like Adam. Additionally, while the theory is general, empirical validation has focused on computer vision tasks. Future work is required to extend these laws to Transformer architectures and Large Language Models (LLMs), where the computational savings from efficient scaling could be most impactful. Finally, the "cube root" rule for effective parameters remains a heuristic that may vary across different architectural paradigms.

Summary updated

Abstract

As neural networks continue to grow in size but datasets might not, it is vital to understand how much performance improvement can be expected: is it more important to scale network size or data volume? Thus, neural network scaling laws, which characterize how test error varies with network size and data volume, have become increasingly important. However, existing scaling laws are often applicable only in limited regimes and often do not incorporate or predict well-known phenomena such as double descent. Here, we present a novel theoretical characterization of how three factors --- model size, training time, and data volume --- interact to determine the performance of deep neural networks. We first establish a theoretical and empirical equivalence between scaling the size of a neural network and increasing its training time proportionally. Scale-time equivalence challenges the current practice, wherein large models are trained for small durations, and suggests that smaller models trained over extended periods could match their efficacy. It also leads to a novel method for predicting the performance of large-scale networks from small-scale networks trained for extended epochs, and vice versa. We next combine scale-time equivalence with a linear model analysis of double descent to obtain a unified theoretical scaling law, which we confirm with experiments across vision benchmarks and network architectures. These laws explain several previously unexplained phenomena: reduced data requirements for generalization in larger models, heightened sensitivity to label noise in overparameterized models, and instances where increasing model scale does not necessarily enhance performance. Our findings hold significant implications for the practical deployment of neural networks, offering a more accessible and efficient path to training and fine-tuning large models.
BibTeX
@misc{boopathy2024unified,
  author = {Akhilan Boopathy and Ila Fiete},
  title = {{Unified Neural Network Scaling Laws and Scale-time Equivalence}},
  year = {2024},
  eprint = {2409.05782},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.05782},
}
Figure