Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition

Ben Adlam, Jeffrey Pennington

Abstract & TL;DR

Classical bias-variance trade-offs fail to explain the "double descent" phenomenon observed in modern neural networks, where highly overparameterized models generalize remarkably well. This paper introduces a novel, symmetric bias-variance decomposition that separates the effects of data sampling, model initialization, and label noise. By applying this framework to random feature regression, the authors prove that the infamous test error spike at the interpolation threshold is driven by the interaction between sampling and initialization variance. Crucially, this interaction can be elegantly mitigated through ensembling and bagging.

The Problem: Ambiguity in Standard Decompositions

In classical machine learning, the bias-variance trade-off dictates that models of intermediate complexity generalize best. Too simple implies high bias; too complex implies high variance. However, modern deep learning defies this logic. Highly overparameterized models that perfectly fit the training data often generalize remarkably well, following a "double descent" curve where test error spikes at the interpolation threshold (where the model just barely fits the data) and then drops again as capacity increases.

Previous attempts to explain this phenomenon using standard bias-variance decompositions have fallen short. Modern machine learning involves multiple sources of randomness—specifically, the sampling of the training data, the initialization of the model parameters, and noise in the labels. Prior works attempted to use the Law of Total Variance to isolate these sources, but this requires choosing an arbitrary order of conditioning (e.g., conditioning on data first, then parameters, or vice versa). This ambiguity led to conflicting conclusions in the literature about whether "variance due to sampling" or "variance due to optimization" was responsible for the double descent spike.

Methods: A Symmetric Variance Decomposition

To resolve this ambiguity, the authors introduce a symmetric variance decomposition (conceptually related to ANOVA) that does not rely on an arbitrary order of conditioning. Instead of merely splitting variance into "explained by A" and "unexplained by A," they break the total variance into main effects (e.g., variance strictly from the data, variance strictly from the parameters) and interaction effects (e.g., variance from their combination).

Venn diagrams of variance decompositions and their corresponding high-dimensional asymptotic behaviors.

To ground this theoretically, the authors perform a high-dimensional asymptotic analysis of random feature kernel regression, a well-known proxy for wide neural networks. Using tools from random matrix theory and operator-valued free probability, they derive exact, closed-form analytical expressions for the bias and all individual variance components as the dataset size, feature dimensionality, and hidden layer size all approach infinity at a constant ratio.

Key Insights

  • Bias is strictly monotonic: Unlike classical theory, which suggests bias and variance trade off against one another, the authors prove that in this regime, bias strictly decreases as model width increases. Bias plays no role in the double descent error spike.
  • Interaction causes the divergence: The spike in test error at the interpolation boundary is not caused by the data alone, nor the initialization alone. It is entirely driven by the interaction variance between the training data sample and the model initialization.
  • Label noise amplifies, but doesn't cause, double descent: The test loss diverges at the interpolation threshold even with perfectly clean labels. However, label noise adds its own diverging interaction term, which explains recent empirical observations that label noise exacerbates the double descent effect.
  • Resolving past theoretical conflicts: By showing that the interaction term is the culprit, the authors explain why previous papers reached contradictory conclusions. Forcing a two-part decomposition using the Law of Total Variance lumps the interaction term into whichever variable was not conditioned on, leading to false attribution.

Key Results: Ensembling, Bagging, and Compute Allocation

The authors successfully derived exact polynomial equations for the bias and all seven variance components in the high-dimensional limit. They mathematically proved that at the interpolation boundary, the interaction terms diverge to infinity, while all other terms (including bias and main-effect variances) remain finite.

Because the divergence is caused by interaction terms, the authors show that ensembling (averaging predictions over multiple initializations) and bagging (averaging over multiple data samples) directly suppress these specific diverging terms.

Effects of ensembling and bagging on test error, and the optimal ratio of bagging to ensembling.

Furthermore, they provide a closed-form equation for the optimal ratio of bagging to ensembling given a fixed computational budget. For their specific kernel regression problem, ensembling is more efficient at small widths, while bagging is more efficient at large widths.

Limitations and Future Work

The exact mathematical results are derived for a specific theoretical setting (random feature kernel regression with Gaussian data in the high-dimensional limit). While this is a standard and highly useful proxy for studying deep learning dynamics, it remains a simplification of real-world, multi-layer neural networks trained with SGD on structured data like images or text.

Looking forward, this fine-grained understanding of variance sources can be applied practically. By identifying which sources of variance dominate a given model architecture or dataset, practitioners can make mathematically informed decisions about exactly which types of ensemble or bagging techniques will yield the best performance for their specific computational budget.

Summary updated

Abstract

Classical learning theory suggests that the optimal generalization performance of a machine learning model should occur at an intermediate model complexity, with simpler models exhibiting high bias and more complex models exhibiting high variance of the predictive function. However, such a simple trade-off does not adequately describe deep learning models that simultaneously attain low bias and variance in the heavily overparameterized regime. A primary obstacle in explaining this behavior is that deep learning algorithms typically involve multiple sources of randomness whose individual contributions are not visible in the total variance. To enable fine-grained analysis, we describe an interpretable, symmetric decomposition of the variance into terms associated with the randomness from sampling, initialization, and the labels. Moreover, we compute the high-dimensional asymptotic behavior of this decomposition for random feature kernel regression, and analyze the strikingly rich phenomenology that arises. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior and can diverge at the interpolation boundary, even in the absence of label noise. The divergence is caused by the interaction between sampling and initialization and can therefore be eliminated by marginalizing over samples (i.e. bagging) or over the initial parameters (i.e. ensemble learning).
BibTeX
@inproceedings{adlam2020understanding,
  author = {Ben Adlam and Jeffrey Pennington},
  title = {{Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=HYWbOgP4Se5},
}
Figure