Abstract: This paper explores the "suspicious alignment" phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. During the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease, subsequently entering a rising phase and stabilizing at a high-alignment equilibrium. Paradoxically, updates along this highly-aligned dominant subspace prove ineffective at reducing loss. This work provides a fine-grained analysis in a high-dimensional quadratic setup, revealing how step size selection produces this phenomenon. We propose an adaptive critical step size $\eta_t^*$ that separates alignment-decreasing from alignment-increasing regimes and prove that under sufficient ill-conditioning, a step size interval exists where bulk-space updates decrease loss while dominant-space updates increase it.
The Paradox of Suspicious Alignment
Deep neural networks are often optimized over ill-conditioned loss landscapes, frequently described as a "river-valley" structure: a few narrow, steep directions of high curvature (the valley walls) and a vast number of flat, low-curvature directions (the river).
Previous research observed a strange phenomenon: as training progresses, SGD gradients become almost entirely aligned with the steep, dominant subspace. Intuitively, optimizing along these highly aligned directions should be the key to efficiency. However, recent empirical findings showed the opposite: in this high-alignment regime, updating the model only along the dominant subspace fails to decrease training loss, while updating along the seemingly insignificant flat "bulk" subspace successfully reduces it. This paper seeks to mathematically explain how step size governs this alignment and why dominant-subspace updates fail.
Theoretical Framework
To isolate these mechanics, the authors utilize a high-dimensional quadratic optimization model ($d \to \infty$). They perform a spectral decomposition of the Hessian matrix into two blocks separated by a distinct eigenvalue gap:
- Dominant Block ($\mathcal{D}$): The top $k$ largest eigenvalues representing high curvature.
- Bulk Block ($\mathcal{B}$): The remaining $d-k$ smaller eigenvalues representing low curvature.
The analysis tracks an alignment metric ($\theta_t$), measuring the proportion of the gradient's norm residing in the dominant subspace. By conducting a probabilistic, asymptotic analysis of SGD with noise, the authors track how $\theta_t$ evolves step-by-step.
The Adaptive Step-Size Theory
The core contribution of this work is the discovery of an adaptive step-size threshold, $\eta_t^*$, and the characterization of two distinct alignment regimes:
- Low-Alignment Regime: If the step size $\eta_t$ is smaller than $\eta_t^*$, alignment decreases (the optimizer moves toward the "river"). If it is larger than $\eta_t^*$, alignment increases (the optimizer climbs the "valley" walls).
- High-Alignment Regime: There is a "ceiling" to this alignment. If alignment exceeds a computable threshold $\theta_t^*$, it becomes self-correcting; the expected alignment will decrease regardless of the step size used.
Resolving the Paradox
The paper explains the failure of dominant updates through "Stability Disparity." The maximum stable step size for the dominant subspace is significantly smaller than the threshold for the bulk subspace. Under standard training conditions in the high-alignment regime, the step size typically falls between these two thresholds. Consequently, the step size is too large for the dominant space (causing the loss to bounce and increase) but small enough for the bulk space (allowing the loss to safely decrease).
Furthermore, the authors prove that for a constant step size and large initialization, SGD exhibits a distinct two-phase behavior: an initial transient phase where alignment monotonically drops, followed by a late phase where alignment rises and stabilizes at a high equilibrium dictated by the noise and the Hessian's spectrum.
Conclusion and Limitations
Numerical simulations (up to $d=500$) perfectly matched these theoretical predictions, confirming that larger spectral gaps lead to higher, more stable final alignment states. While the reliance on a quadratic loss function is a limitation—as real-world neural network landscapes are non-convex—this model provides a powerful tool for isolating optimization mechanics. Future work may extend this analysis to adaptive preconditioning methods like Adam to see how they interact with or bypass suspicious alignment.
Summary updated