Introduction
In recent years, the machine learning community has been fascinated by the double descent phenomenon. Traditionally, statistical learning theory suggested a bias-variance tradeoff where increasing model complexity beyond a certain point leads to overfitting. However, deep learning practice reveals that if parameters are added past the "interpolation threshold"—the point where the model perfectly fits training data—generalization error often starts decreasing again.
While well-documented in supervised regression, the presence of double descent in unsupervised tasks like dimensionality reduction remained unclear. This paper investigates whether subspace fitting exhibits double descent and identifies the specific roles of supervision and model constraints in triggering this behavior.
The Supervision-Orthonormality Plane
The authors bridge the gap between unsupervised subspace fitting (e.g., PCA) and supervised regression by mapping them onto a unified supervision-orthonormality plane. This framework allows for a continuous transition between traditionally distinct learning paradigms.
As shown in Figure 1, the plane is defined by two primary axes:
- Supervision Axis: Ranges from unsupervised (only inputs $x$ are known) to fully supervised (input-output pairs $(x, z)$ are known).
- Orthonormality Axis: Ranges from hard constraints (orthonormal columns, as in standard PCA) to no constraints (standard linear regression).
Methodology
To explore this landscape, the researchers developed several key tools:
- Soft Orthonormality Constraints: They introduced a parameter $\alpha$ to control constraint "softness" via singular value thresholding during optimization.
- Projected Gradient Descent (PGD): This iterative approach was used to solve optimization problems across the plane where closed-form solutions are unavailable.
- Controlled Overparameterization: By varying the number of features ($p$) relative to training samples ($n$), they observed error behavior in the overparameterized regime ($p > n$).
The study utilized synthetic linear data models ($x = U_m z + \epsilon$) to isolate these effects, evaluating performance across 1,000 out-of-sample realizations.
Key Findings: When Does Double Descent Emerge?
The research reveals that double descent is not a universal property of learning but rather a conditional one.
Supervision as a Catalyst
Unsupervised learning appears "immune" to double descent. In standard PCA, generalization error decreases monotonically as more features are added. The characteristic "peak" at the interpolation threshold only emerges as the level of supervision increases. This peak is a byproduct of the model attempting to map specific inputs to outputs without sufficient structural guidance.
Constraints as a Stabilizer
Even in fully supervised settings, structural constraints play a critical role. Enforcing strict orthonormality (hard constraints) can eliminate the double descent peak, reverting the error curve to a monotonic "single descent."
As illustrated in Figure 5, softening the orthonormality constraint (increasing $\alpha$) allows the model to achieve zero in-sample error (interpolation) more easily but simultaneously causes the out-of-sample error peak to become more pronounced at the interpolation boundary ($p \approx n$).
Conclusion
This work demonstrates that double descent is a phenomenon driven by the interaction of supervision and model flexibility. By moving diagonally across the supervision-orthonormality plane—increasing supervision while relaxing constraints—one can observe the transition from stable, monotonic generalization to the complex double descent curves characteristic of modern overparameterized models. These insights provide a clearer theoretical baseline for understanding generalization in both linear and, potentially, non-linear deep learning architectures.
Summary updated