Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis

Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, Yaoqing Yang

TL;DR

During the optimization of deep neural networks, Stochastic Gradient Descent (SGD) gradients often align heavily with the sharpest directions of the loss landscape. Paradoxically, updates along these highly aligned directions fail to reduce the loss effectively—a phenomenon known as "suspicious alignment." This paper provides a rigorous mathematical analysis using a high-dimensional quadratic model to show exactly how step-size selection and landscape ill-conditioning drive this behavior.

What Problem Does This Paper Address?

Deep neural networks are typically optimized over highly ill-conditioned loss landscapes often described as "river valleys": they are steep along a few dominant directions (high curvature) but nearly flat across a massive subspace of bulk directions (low curvature).

As training progresses, SGD gradients exhibit a "tiny subspace effect," becoming increasingly aligned with the dominant subspace. Intuitively, focusing optimization on the most curved directions sounds like an efficiency win. However, recent empirical studies revealed a baffling paradox: in this high-alignment regime, SGD updates projected onto the dominant subspace often fail to decrease the training loss, while updates along the flat bulk subspace—despite carrying negligible gradient norm—successfully drive the loss down.

This paper provides complete analytical answers to two fundamental questions:

  1. How does the step size ($\eta_t$) govern the evolution of gradient alignment?
  2. Why does Dominant Projected SGD (DSGD) fail to reduce loss while Bulk Projected SGD (BSGD) succeeds under high alignment?

Methods

The authors analyze the dynamics of SGD within a high-dimensional quadratic optimization framework ($d \to \infty$) under arbitrary finite second-moment noise. They track the squared alignment function ($\theta_t$), which measures the proportion of the gradient's norm residing within the dominant eigenspace.

To dissect the alignment trajectory, they introduce a comparison functional that tracks the exact one-step expected change in alignment. They also analyze two idealized algorithmic variants to isolate the behavior of different landscape components:

  • Dominant Projected SGD (DSGD): Updates are strictly projected onto the high-curvature subspace.
  • Bulk Projected SGD (BSGD): Updates are strictly projected onto the flat bulk subspace.

Datasets

No real-world datasets are used in this work. The paper relies entirely on synthetic numerical simulations to validate its theoretical framework. The experiments utilize randomly initialized positive-definite symmetric matrices to represent the Hessian ($A \in \mathbb{R}^{d \times d}$), fixing the total dimension at $d = 500$ and the dominant subspace dimension at $k = 50$. The models are evaluated across various spectral gaps ($m = \lambda_k / \lambda_{k+1}$).

Key Insights and Ideas

  • The Critical Step-Size Threshold ($\eta_t^*$): In low-alignment regimes, the direction of alignment evolution depends entirely on the step size. Small step sizes ($\eta_t < \eta_t^*$) cause alignment to decrease, pushing the parameters closer to the "river bed," while large step sizes ($\eta_t > \eta_t^*$) cause alignment to increase.
  • Self-Correction at High Alignment: Once the gradient alignment crosses a high threshold, it becomes self-correcting. In this regime, the expected alignment decreases for any positive step size, preventing the optimization from getting permanently stuck at maximum alignment.
Low-alignment vs High-alignment regimes
  • Resolving the Paradox via Stability Disparity: The authors prove that under sufficient ill-conditioning, a specific step-size interval exists where DSGD increases the loss while BSGD decreases it. This mathematically explains why projecting updates onto the dominant subspace is ineffective during high alignment.
  • Distinct Two-Phase Dynamics: For a constant step size and large initialization, SGD naturally undergoes a two-phase journey: an initial transient phase where alignment monotonically decreases (tracking the river), followed by a rising phase that stabilizes at a high-alignment equilibrium driven by gradient noise.
Two-phase SGD dynamic showing alignment trajectory

Key Results

  • Asymptotic Limits: The authors prove that as the spectral gap grows infinitely large ($m \to \infty$), the critical alignment threshold ($\theta^{\text{crit}}$) converges to $1$. This means the unstable regime where dominant updates increase the loss expands to occupy nearly the entire alignment space $[0, 1)$.
  • Simulation Alignment: Numerical simulations across multiple random seeds confirm the theoretical two-phase behavior. As the spectral gap ($m$) increases from 5 to 500, the stable region of the alignment function tightly approaches $1$, matching the predictions of their late-phase equilibrium theorem.

Limitations and Future Work

  • Quadratic Simplification: The core limitation is that the fine-grained analysis is conducted within a quadratic loss setup. While deep network loss landscapes locally resemble quadratics, extending these strict proofs to highly non-convex, non-quadratic regimes remains an open challenge.
  • Simplified Noise Assumptions: The noise model, while allowing arbitrary covariance, is analyzed primarily under standard Gaussian assumptions, which may omit complex state-dependent noise structures found in real-world mini-batch training.
  • Extension to Preconditioners: The authors note that exploring how step-size selection influences alignment in advanced preconditioning methods (like Adam or Shampoo) is an important direction for future research.

Summary updated

Abstract

This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. This phenomenon describes the behavior of gradient alignment in SGD updates. Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease. Subsequently, it enters a rising phase and eventually stabilizes in a high-alignment phase. The alignment is considered ``suspicious'' because, paradoxically, the projected gradient update along this highly-aligned dominant subspace proves ineffective at reducing the loss. The focus of this work is to give a fine-grained analysis in a high-dimensional quadratic setup about how step size selection produces this phenomenon. Our main contribution can be summarized as follows: We propose a step-size condition revealing that in low-alignment regimes, an adaptive critical step size $η_t^*$ separates alignment-decreasing ($η_t < η_t^*$) from alignment-increasing ($η_t > η_t^*$) regimes, whereas in high-alignment regimes, the alignment is self-correcting and decreases regardless of the step size. We further show that under sufficient ill-conditioning, a step size interval exists where projecting the SGD updates to the bulk space decreases the loss while projecting them to the dominant space increases the loss, which explains a recent empirical observation that projecting gradient updates to the dominant subspace is ineffective. Finally, based on this adaptive step-size theory, we prove that for a constant step size and large initialization, SGD exhibits this distinct two-phase behavior: an initial alignment-decreasing phase, followed by stabilization at high alignment.
BibTeX
@inproceedings{deng2026suspicious,
  author = {Shenyang Deng and Boyao Liao and Zhuoli Ouyang and Tianyu Pang and Minhak Song and Yaoqing Yang},
  title = {{Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis}},
  booktitle = {ALT},
  year = {2026},
  url = {https://openreview.net/forum?id=TACb1ONsU5},
}
Figure