Abstract
Classical learning theory suggests that the optimal generalization performance of a machine learning model should occur at an intermediate model complexity, with simpler models exhibiting high bias and more complex models exhibiting high variance of the predictive function. However, such a simple trade-off does not adequately describe deep learning models that simultaneously attain low bias and variance in the heavily overparameterized regime. A primary obstacle in explaining this behavior is that deep learning algorithms typically involve multiple sources of randomness whose individual contributions are not visible in the total variance. To enable fine-grained analysis, we describe an interpretable, symmetric decomposition of the variance into terms associated with the randomness from sampling, initialization, and the labels. Moreover, we compute the high-dimensional asymptotic behavior of this decomposition for random feature kernel regression, and analyze the strikingly rich phenomenology that arises. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior and can diverge at the interpolation boundary, even in the absence of label noise. The divergence is caused by the interaction between sampling and initialization and can therefore be eliminated by marginalizing over samples (i.e. bagging) or over the initial parameters (i.e. ensemble learning).
Introduction
The classical bias-variance tradeoff posits that as model complexity increases, bias decreases while variance increases, leading to a U-shaped test error curve. However, modern deep learning models often exhibit "double descent," where the test error decreases again in the overparameterized regime. Explaining this phenomenon requires moving beyond standard "blunt" variance decompositions. This research introduces a symmetric variance decomposition to isolate the contributions of data sampling ($X$), model initialization ($P$), and label noise ($\epsilon$).
A Fine-Grained Variance Decomposition
To analyze the error components, the authors employ a functional ANOVA approach. This provides a symmetric framework where all sources of randomness are treated equally, unlike traditional asymmetric "law of total variance" applications. The study focuses on Random Feature Kernel Regression, using tools from Random Matrix Theory and operator-valued free probability to derive exact formulas for error terms in the high-dimensional limit.
The Interaction Spike
A key insight is that the bias component decreases monotonically as the network width increases. The characteristic "spike" in double descent is purely a variance phenomenon occurring at the interpolation boundary (where the number of parameters equals the number of data points). Specifically, the spike is driven by the interaction between sampling and initialization ($V_{PX}$) and their interaction with label noise ($V_{PX\epsilon}$). As shown in Figure 1, these terms diverge at the interpolation threshold, even in the absence of label noise.
Mitigation through Ensembling and Bagging
Since the generalization spike is rooted in specific interaction terms, it can be mitigated by marginalizing out the sources of randomness. Ensembling (averaging over multiple initializations $P$) and bagging (averaging over data samples $X$) effectively reduce these interaction variances.
The analysis reveals an efficiency tradeoff: for a fixed computational budget, ensembling is more effective for underparameterized models, whereas bagging provides greater error reduction in the overparameterized regime.
Conclusion
This work provides a theoretical microscope for understanding double descent by identifying the interaction between sampling and initialization as the primary driver of the interpolation spike. While the analysis is currently limited to the asymptotic regime of kernel methods (the "lazy training" regime), it offers a rigorous foundation for future studies into the feature-learning dynamics of deep neural networks.
Summary updated