Early Stopping in Deep Networks: Double Descent and How to Eliminate it

Reinhard Heckel, Fatih Furkan Yilmaz

Abstract

Over-parameterized models, such as large deep networks, often exhibit a double descent phenomenon, where as a function of model size, error first decreases, increases, and decreases at last. This intriguing double descent behavior also occurs as a function of training epochs and has been conjectured to arise because training epochs control the model complexity. In this paper, we show that such epoch-wise double descent arises for a different reason: It is caused by a superposition of two or more bias-variance tradeoffs that arise because different parts of the network are learned at different epochs, and eliminating this by proper scaling of stepsizes can significantly improve the early stopping performance. We show this analytically for i) linear regression, where differently scaled features give rise to a superposition of bias-variance tradeoffs, and for ii) a two-layer neural network, where the first and second layer each govern a bias-variance tradeoff. Inspired by this theory, we study two standard convolutional networks empirically and show that eliminating epoch-wise double descent through adjusting stepsizes of different layers improves the early stopping performance significantly.

Introduction: The Mystery of Epoch-wise Double Descent

In classical machine learning, we expect a U-shaped test error curve: as a model trains, bias decreases and variance increases, leading to an optimal "sweet spot" for early stopping. However, modern deep networks often exhibit double descent, where the test error drops, rises (forming a "hump"), and then drops again during training.

Test error showing epoch-wise double descent with a characteristic "hump" before the final descent.

While previous research suggested this was tied to model complexity, this paper tackles epoch-wise double descent—the phenomenon where this behavior happens over the course of training time. This makes choosing an early stopping point incredibly difficult and suggests that our models are performing sub-optimally. The authors ask: Why does this happen, and can we design a training regime that avoids it?

The Superposition Principle

The core insight of the paper is that double descent is essentially a timing issue. If different components of a network (like different layers or features with different scales) learn at different speeds, their individual U-shaped bias-variance curves are offset in time. When these curves are summed to form the total risk, the peak of one and the valley of another create the "double descent" hump.

The authors provide theoretical proofs for this in linear regression and two-layer neural networks. They show that in many CNNs, the final fully connected layer (the "head") learns much faster than the convolutional layers (the "features"). This mismatch is a primary driver of double descent in deep learning.

Eliminating the Hump via Step-size Adaptation

Based on their theory, the authors propose a mitigation strategy: scaling the learning rates (stepsizes) of different layers so that their individual "minima" coincide in time. By slowing down the components that learn too fast, the overlapping bias-variance tradeoffs can be aligned, effectively merging multiple U-curves into one.

Comparison of risk curves with constant stepsizes versus adjusted stepsizes to eliminate double descent.

As shown in the figure above, using a constant stepsize (a) leads to a superposition of tradeoffs that creates a hump. By using different stepsizes (b), these tradeoffs are aligned, eliminating the hump and resulting in a lower overall minimum risk (c).

Empirical Validation and Key Results

The study focuses on image classification under noisy conditions (CIFAR-10 with 20% random label noise), as double descent is most prominent when models are forced to "memorize" noise.

  • Linear Regression: The authors successfully demonstrated that choosing stepsizes according to a specific formula could align feature learning and eliminate the double descent curve.
  • CNN Architectures: On a 5-layer CNN and ResNet-18, decreasing the learning rate of the final fully connected layer relative to the convolutional layers eliminated the test error "hump" and achieved better final test error than standard training.
  • Bias-Variance Decomposition: Numerical analysis showed that the "bias" term itself can exhibit a double-descent shape, while the "variance" term increases monotonically, providing a new perspective on these dynamics.

Conclusion and Future Work

Epoch-wise double descent is not an inherent mystery but a consequence of misaligned learning dynamics. By adjusting stepsizes, researchers can eliminate this behavior and significantly improve early stopping performance.

Future work includes automating this process. While stepsizes were adjusted manually here based on layer roles, future algorithms could dynamically sense learning speeds to keep tradeoffs aligned. Additionally, exploring how these dynamics apply outside the "lazy training" regime and with different types of data noise remains an open area of research.

Summary updated

Abstract

Over-parameterized models, such as large deep networks, often exhibit a double descent phenomenon, whereas a function of model size, error first decreases, increases, and decreases at last. This intriguing double descent behavior also occurs as a function of training epochs and has been conjectured to arise because training epochs control the model complexity. In this paper, we show that such epoch-wise double descent arises for a different reason: It is caused by a superposition of two or more bias-variance tradeoffs that arise because different parts of the network are learned at different epochs, and eliminating this by proper scaling of stepsizes can significantly improve the early stopping performance. We show this analytically for i) linear regression, where differently scaled features give rise to a superposition of bias-variance tradeoffs, and for ii) a two-layer neural network, where the first and second layer each govern a bias-variance tradeoff. Inspired by this theory, we study two standard convolutional networks empirically and show that eliminating epoch-wise double descent through adjusting stepsizes of different layers improves the early stopping performance significantly.
BibTeX
@inproceedings{heckel2021early,
  author = {Reinhard Heckel and Fatih Furkan Yilmaz},
  title = {{Early Stopping in Deep Networks: Double Descent and How to Eliminate it}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=tlV90jvZbw},
}
Figure