Abstract
Modern machine learning models often exhibit "double descent," where predictive risk peaks at the interpolation threshold before decreasing in the over-parameterized regime. This phenomenon challenges the classical bias-variance trade-off. This paper introduces the Generalized Aliasing Decomposition (GAD), a mathematical framework that partitions predictive risk into model insufficiency, data insufficiency, and generalized aliasing. By identifying an "Aliasing Operator" that depends only on the model class and sampling points, GAD provides a label-independent method to predict and control generalization error. We demonstrate that the dramatic error peaks at the interpolation threshold are caused by the aliasing of unmodeled signals, offering a rigorous explanation for non-monotonic risk curves across various domains.
Introduction: The Mystery of Double Descent
In classical statistics, the bias-variance trade-off is the primary lens for understanding generalization: as model complexity increases, bias decreases but variance increases. This suggests a "sweet spot" where complexity is balanced to avoid overfitting. However, modern deep learning has shattered this intuition. We frequently observe double descent, where error hits a peak at the "interpolation threshold" (where the number of parameters equals the number of data points) and then falls as the model becomes over-parameterized.
The classical lens fails to explain these non-monotonic, "bumpy" error curves. This paper seeks to replace that outdated framework with a rigorous decomposition that accounts for the structured interference inherent in modern high-dimensional modeling.
Methods: Generalized Aliasing Decomposition (GAD)
The authors propose the Generalized Aliasing Decomposition (GAD) by partitioning the function space into a "modeled" subspace ($\mathcal{M}$) and an "unmodeled" subspace ($\mathcal{U}$). The core of the method is the Aliasing Operator ($A$): $$A = M_{\mathcal{T}\mathcal{M}}^+ M_{\mathcal{T}\mathcal{U}}$$ This operator quantifies how signals from the unmodeled space (high-frequency "noise" or details) are "aliased" or redirected into the modeled parameters due to the specific sampling of the data.
The GAD framework breaks total risk into three distinct, intuitive components:
- Model Insufficiency: Error caused by the model being too simple to represent the true signal (analogous to classical bias).
- Data Insufficiency: Error arising because there isn't enough data to uniquely determine parameters, which dominates the over-parameterized regime.
- Generalized Aliasing: The "interference" term that causes dramatic peaks in error curves by explaining how unmodeled components corrupt inferred parameters.
Key Insights and Results
Label-Independent Risk Prediction
Perhaps the most powerful insight is that the Aliasing Operator ($A$) depends only on the model class and the sampling points, not the data labels ($y$). As shown in Figure 2, the "Aliasing Norm" tracks the actual predictive risk remarkably well. This means researchers can calculate the risk of aliasing and optimize experimental designs before collecting or labeling a single data point.
Explaining the Interpolation Peak
The peak in double descent occurs because the design matrix becomes nearly singular at the interpolation threshold, causing the aliasing operator's norm to explode. This provides a geometric explanation for the "variance explosion" often cited in classical literature.
Empirical Validation
The authors validated GAD across diverse benchmarks:
- Machine Learning: MNIST and CIFAR-10 using Random ReLU and Fourier feature models.
- Physical Systems: Synthetic datasets for string displacement and materials science (Pt-Cu alloy formation enthalpy).
- FFT Stability: The paper explains the robustness of the Fast Fourier Transform (FFT); on a uniform grid, the aliasing operator has a constant unit norm, preventing the variance peaks seen in other models.
- Regularization: $L_2$ regularization (Ridge regression) suppresses the aliasing peak by bounding the norm of the pseudoinverse, effectively smoothing the double descent curve.
Conclusion and Future Work
GAD offers a new path for understanding generalization in the over-parameterized regime. While currently focused on linear models and local linearizations (like Neural Tangent Kernels), the framework provides actionable insights for active learning and experimental design. By choosing training points that minimize the aliasing operator's norm, researchers can design models that are inherently more stable and generalize better even before seeing the labels.
Summary updated