Optimal Regularization Can Mitigate Double Descent

Preetum Nakkiran, Prayaag Venkat, Sham M. Kakade, Tengyu Ma

Abstract: Recent empirical and theoretical studies have shown that many learning algorithms -- from linear regression to neural networks -- can have test performance that is non-monotonic in quantities such the sample size and model size. This striking phenomenon, often referred to as "double descent", has raised questions of if we need to re-think our current understanding of generalization. In this work, we study whether the double-descent phenomenon can be avoided by using optimal regularization. Theoretically, we prove that for certain linear regression models with isotropic data distribution, optimally-tuned $\ell_2$ regularization achieves monotonic test performance as we grow either the sample size or the model size. We also demonstrate empirically that optimally-tuned $\ell_2$ regularization can mitigate double descent for more general models, including neural networks. Our results suggest that it may also be informative to study the test risk scalings of various algorithms in the context of appropriately tuned regularization.

Introduction: The Challenge of Non-Monotonic Generalization

In classical statistics, the bias-variance tradeoff suggests that as model complexity increases, bias decreases while variance eventually explodes, resulting in a U-shaped risk curve. However, modern deep learning often exhibits "double descent." As model size increases, test error initially follows the U-shape but then reaches a sharp peak at the "interpolation threshold" (where the model perfectly fits the training data) before decreasing again in the overparameterized regime.

A similar, even more counter-intuitive "sample-wise double descent" exists, where giving a model more training data can actually hurt its test performance. This paper asks a fundamental question: Is double descent an unavoidable feature of modern machine learning, or is it just a consequence of how we regularize (or fail to regularize) our models?

Methodology

The authors employ a combination of non-asymptotic theory and empirical stress tests to address this question:

  1. Non-Asymptotic Theory: Unlike previous asymptotic analyses, this work provides proofs for fixed sample sizes ($n$) and model dimensions ($d$), focusing on Ridge Regression (linear regression with an $\ell_2$ penalty).
  2. Singular Value Interlacing: Using the Cauchy interlacing theorem, the authors compare the singular values of data matrices with varying sample sizes to prove that the expected risk of an optimally regularized model decreases as more data is added.
  3. Optimal Tuning: The core argument is that the regularization parameter $\lambda$ must be tuned to minimize the expected test risk for every specific configuration of $n$ and $d$.
  4. Empirical Validation: The theory is extended to non-linear regimes using Random ReLU Features on Fashion-MNIST and Convolutional Neural Networks (CNNs) on CIFAR-100.

Key Insights: Regularization as a "Peak-Smoother"

The "peak" in double descent typically occurs because model variance explodes near the interpolation threshold ($n \approx d$). Optimal regularization effectively penalizes this variance, smoothing the peak into a monotonic curve.

Expected excess test risk as a function of sample size for various regularization strengths. While unregularized models (λ=0) exhibit a sharp double descent peak, optimal regularization (λ=λ_opt) results in a monotonic decrease in risk.

As shown in the figure above, as the regularization parameter $\lambda$ approaches the optimal value $\lambda_{opt}$, the non-monotonic behavior (the "bump") disappears. This confirms the intuition that with proper regularization, "more is better"—both in terms of data and model capacity. In the simplest isotropic cases, the optimal $\lambda$ is a constant depending only on noise level and signal strength, independent of sample size.

Key Results and Empirical Evidence

  • Mathematical Proofs: Theorems 1 and 3 prove that for well-specified linear models with isotropic data, optimally-tuned ridge regression risk is monotonic in both sample size and model size.
  • Neural Network Mitigation: Experiments on CIFAR-100 demonstrate that while unregularized CNNs show a clear double descent peak as width increases, choosing the optimal weight decay for each width restores near-perfect monotonicity.
  • Triple Descent: In non-isotropic settings where features have varying importance, unregularized models can exhibit multiple peaks ("triple descent"). Optimal regularization successfully eliminates these artifacts as well.

Limitations and Future Work

While compelling, the research notes several challenges:

  • Finding "Optimal" is Hard: Finding the "optimal" $\lambda$ is difficult in practice, as it often requires validation sets which are themselves finite and noisy. Under-regularized models remain the most prone to double descent.
  • Linear vs. Non-Linear: Formal proofs are currently limited to linear models. Extending this non-asymptotic theory to deep, non-linear architectures remains a major open challenge.
  • Data-Dependent Regularization: For non-isotropic data, the ideal regularizer might require knowledge of the data covariance, suggesting that unlabeled data could play a role in designing better regularizers for labeled tasks.

Summary updated

Abstract

Recent empirical and theoretical studies have shown that many learning algorithms -- from linear regression to neural networks -- can have test performance that is non-monotonic in quantities such the sample size and model size. This striking phenomenon, often referred to as "double descent", has raised questions of if we need to re-think our current understanding of generalization. In this work, we study whether the double-descent phenomenon can be avoided by using optimal regularization. Theoretically, we prove that for certain linear regression models with isotropic data distribution, optimally-tuned $\ell_2$ regularization achieves monotonic test performance as we grow either the sample size or the model size. We also demonstrate empirically that optimally-tuned $\ell_2$ regularization can mitigate double descent for more general models, including neural networks. Our results suggest that it may also be informative to study the test risk scalings of various algorithms in the context of appropriately tuned regularization.
BibTeX
@inproceedings{nakkiran2021optimal,
  author = {Preetum Nakkiran and Prayaag Venkat and Sham M. Kakade and Tengyu Ma},
  title = {{Optimal Regularization Can Mitigate Double Descent}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=7R7fAoUygoa},
}
Figure