Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis

Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, Yaoqing Yang

Abstract: This paper explores the "suspicious alignment" phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. During the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease, subsequently entering a rising phase and stabilizing at a high-alignment equilibrium. Paradoxically, updates along this highly-aligned dominant subspace prove ineffective at reducing loss. This work provides a fine-grained analysis in a high-dimensional quadratic setup, revealing how step size selection produces this phenomenon. We propose an adaptive critical step size $\eta_t^*$ that separates alignment-decreasing from alignment-increasing regimes and prove that under sufficient ill-conditioning, a step size interval exists where bulk-space updates decrease loss while dominant-space updates increase it.

The Paradox of Suspicious Alignment

Deep neural networks are often optimized over ill-conditioned loss landscapes, frequently described as a "river-valley" structure: a few narrow, steep directions of high curvature (the valley walls) and a vast number of flat, low-curvature directions (the river).

Previous research observed a strange phenomenon: as training progresses, SGD gradients become almost entirely aligned with the steep, dominant subspace. Intuitively, optimizing along these highly aligned directions should be the key to efficiency. However, recent empirical findings showed the opposite: in this high-alignment regime, updating the model only along the dominant subspace fails to decrease training loss, while updating along the seemingly insignificant flat "bulk" subspace successfully reduces it. This paper seeks to mathematically explain how step size governs this alignment and why dominant-subspace updates fail.

Figure 1: The two-phase evolution of gradient alignment (right) corresponds to the geometric trajectory of the optimizer through dominant and bulk subspaces (left).

Theoretical Framework

To isolate these mechanics, the authors utilize a high-dimensional quadratic optimization model ($d \to \infty$). They perform a spectral decomposition of the Hessian matrix into two blocks separated by a distinct eigenvalue gap:

  • Dominant Block ($\mathcal{D}$): The top $k$ largest eigenvalues representing high curvature.
  • Bulk Block ($\mathcal{B}$): The remaining $d-k$ smaller eigenvalues representing low curvature.

The analysis tracks an alignment metric ($\theta_t$), measuring the proportion of the gradient's norm residing in the dominant subspace. By conducting a probabilistic, asymptotic analysis of SGD with noise, the authors track how $\theta_t$ evolves step-by-step.

The Adaptive Step-Size Theory

The core contribution of this work is the discovery of an adaptive step-size threshold, $\eta_t^*$, and the characterization of two distinct alignment regimes:

  1. Low-Alignment Regime: If the step size $\eta_t$ is smaller than $\eta_t^*$, alignment decreases (the optimizer moves toward the "river"). If it is larger than $\eta_t^*$, alignment increases (the optimizer climbs the "valley" walls).
  2. High-Alignment Regime: There is a "ceiling" to this alignment. If alignment exceeds a computable threshold $\theta_t^*$, it becomes self-correcting; the expected alignment will decrease regardless of the step size used.
Figure 2: Mathematical conditions defining the Low-Alignment and High-Alignment regimes based on the adaptive step size.

Resolving the Paradox

The paper explains the failure of dominant updates through "Stability Disparity." The maximum stable step size for the dominant subspace is significantly smaller than the threshold for the bulk subspace. Under standard training conditions in the high-alignment regime, the step size typically falls between these two thresholds. Consequently, the step size is too large for the dominant space (causing the loss to bounce and increase) but small enough for the bulk space (allowing the loss to safely decrease).

Furthermore, the authors prove that for a constant step size and large initialization, SGD exhibits a distinct two-phase behavior: an initial transient phase where alignment monotonically drops, followed by a late phase where alignment rises and stabilizes at a high equilibrium dictated by the noise and the Hessian's spectrum.

Conclusion and Limitations

Numerical simulations (up to $d=500$) perfectly matched these theoretical predictions, confirming that larger spectral gaps lead to higher, more stable final alignment states. While the reliance on a quadratic loss function is a limitation—as real-world neural network landscapes are non-convex—this model provides a powerful tool for isolating optimization mechanics. Future work may extend this analysis to adaptive preconditioning methods like Adam to see how they interact with or bypass suspicious alignment.

Summary updated

Abstract

This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. This phenomenon describes the behavior of gradient alignment in SGD updates. Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease. Subsequently, it enters a rising phase and eventually stabilizes in a high-alignment phase. The alignment is considered ``suspicious'' because, paradoxically, the projected gradient update along this highly-aligned dominant subspace proves ineffective at reducing the loss. The focus of this work is to give a fine-grained analysis in a high-dimensional quadratic setup about how step size selection produces this phenomenon. Our main contribution can be summarized as follows: We propose a step-size condition revealing that in low-alignment regimes, an adaptive critical step size $η_t^*$ separates alignment-decreasing ($η_t < η_t^*$) from alignment-increasing ($η_t > η_t^*$) regimes, whereas in high-alignment regimes, the alignment is self-correcting and decreases regardless of the step size. We further show that under sufficient ill-conditioning, a step size interval exists where projecting the SGD updates to the bulk space decreases the loss while projecting them to the dominant space increases the loss, which explains a recent empirical observation that projecting gradient updates to the dominant subspace is ineffective. Finally, based on this adaptive step-size theory, we prove that for a constant step size and large initialization, SGD exhibits this distinct two-phase behavior: an initial alignment-decreasing phase, followed by stabilization at high alignment.
BibTeX
@inproceedings{deng2026suspicious,
  author = {Shenyang Deng and Boyao Liao and Zhuoli Ouyang and Tianyu Pang and Minhak Song and Yaoqing Yang},
  title = {{Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis}},
  booktitle = {ALT},
  year = {2026},
  url = {https://openreview.net/forum?id=TACb1ONsU5},
}
Figure