Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability

Eric Regis, Sinho Chewi

TL;DR

Rod Flow is a new continuous-time model for gradient descent (GD) that remains accurate even when training becomes unstable—a regime known as the "Edge of Stability." By treating consecutive GD iterates as the endpoints of a physical "rod," the authors derive a system of ODEs that is more accurate than traditional Gradient Flow, more efficient than previous state-of-the-art models, and theoretically explains how neural networks self-stabilize during training.

What Problem Does This Paper Address?

In classical optimization theory, we assume the learning rate ($\eta$) is small enough that the loss decreases monotonically. Specifically, the "sharpness" (the largest eigenvalue of the Hessian) should stay below $2/\eta$. If it exceeds this threshold, the "descent lemma" breaks, and we expect the optimizer to diverge.

However, modern neural networks often train at the Edge of Stability (EoS). In this regime, sharpness actually rises until it hits $2/\eta$ and then hovers there. The iterates don't diverge; instead, they exhibit rapid, fine-grained oscillations.

Standard Gradient Flow ($\dot{w} = -\nabla L(w)$) fails here because it assumes a smooth, non-oscillatory trajectory. A recent alternative, Central Flow, attempts to model the "average" path of these oscillations but relies on solving a complex optimization problem (a semidefinite complementarity problem) at every step, making it computationally heavy and difficult to analyze mathematically.

Methods

The authors introduce Rod Flow, which models GD iterates not as a single point, but as an extended one-dimensional object—a "rod."

  1. The Physical Picture: The rod is defined by its center ($\bar{w}$) and its half-length ($\delta$). The endpoints of the rod ($\bar{w} + \delta$ and $\bar{w} - \delta$) represent consecutive GD iterates.
  2. Principled Derivation: Instead of heuristic smoothing, the authors write down exact difference equations for how the center and the "extent" ($\Sigma \approx \delta \otimes \delta$) evolve.
  3. Backward Error Analysis (BEA): They use BEA—a technique from numerical analysis—to find the specific ODE that actually passes through these discrete points. This accounts for the discretization error that standard Gradient Flow ignores.
  4. Computational Efficiency: To make this practical for large networks, they represent the covariance matrix $\Sigma$ as a low-rank object (typically rank-3) and use "forward-over-reverse" automatic differentiation to compute Hessian-Vector Products (HVPs) without ever storing a full Hessian matrix.

Datasets

The authors validated Rod Flow using:

  • Toy Problems: A 2D "Square Root Loss" landscape designed to trigger EoS behavior.
  • Neural Networks:
    • A 3-layer MLP on CIFAR-10.
  • A 3-layer CNN on CIFAR-10.
  • A 2-layer Transformer on SST-2.

Key Insights and Ideas

  • Oscillations as Sampling: Rod Flow treats oscillations as a feature, not a bug. By sampling the landscape at the rod's endpoints, the model "sees" the non-linearity of the loss.
  • The Mechanism of Self-Stabilization: The paper proves that in a quartic (fourth-order) potential, if the sharpness gets too high, the "rod" expands. This expansion forces the endpoints into flatter regions of the landscape, which naturally pushes the sharpness back down toward $2/\eta$. This provides a mechanistic explanation for why GD doesn't explode at the EoS.
  • Unifying Models: The authors show that the previous "Central Flow" model is actually a first-order Taylor approximation of Rod Flow. By keeping the higher-order terms, Rod Flow captures the dynamics more accurately, especially in regions with high curvature.

Key Results

  • Superior Accuracy: On toy examples, Rod Flow tracks the true GD trajectory precisely, whereas Central Flow and Gradient Flow deviate significantly. On neural networks, it matches or exceeds Central Flow's accuracy.
  • Massive Speedup: Rod Flow is significantly faster than Central Flow—up to 17x faster as the number of "sharp" directions in the Hessian increases. This is because Rod Flow uses explicit ODE updates rather than solving an optimization problem at each step.
  • Theoretical Consistency: The model correctly recovers the $2/\eta$ stability threshold and explains the "progressive sharpening" and "self-stabilization" phases observed in real-world deep learning.

Limitations and Future Work

  • The "Turbulent" Phase: Rod Flow is designed for the "Steady-State" EoS (Phase III). It cannot currently capture the wild, high-amplitude "catapult" spikes that occur during the initial transition to the EoS (Phase II).
  • Computational Overhead: While much faster than Central Flow, Rod Flow is still roughly 4–6x more expensive than standard GD due to the need for multiple gradient evaluations and HVPs. It is a tool for understanding dynamics, not yet a replacement for standard training.
  • The Origin of Sharpening: While the paper explains how the EoS is maintained, it does not explain the underlying cause of "progressive sharpening"—why the Hessian eigenvalues grow toward $2/\eta$ in the first place. This remains an open question in the field.

Summary updated

Abstract

How can we understand gradient-based training over non-convex landscapes? The edge of stability phenomenon, introduced in Cohen et al. (2021), indicates that the answer is not so simple: namely, gradient descent (GD) with large step sizes often diverges away from the gradient flow. In this regime, the "Central Flow", recently proposed in Cohen et al. (2025), provides an accurate ODE approximation to the GD dynamics over many architectures. In this work, we propose Rod Flow, an alternative ODE approximation, which carries the following advantages: (1) it rests on a principled derivation stemming from a physical picture of GD iterates as an extended one-dimensional object -- a "rod"; (2) it better captures GD dynamics for simple toy examples and matches the accuracy of Central Flow for representative neural network architectures, and (3) is explicit and cheap to compute. Theoretically, we prove that Rod Flow correctly predicts the critical sharpness threshold and explains self-stabilization in quartic potentials. We validate our theory with a range of numerical experiments.
BibTeX
@misc{regis2026rod,
  author = {Eric Regis and Sinho Chewi},
  title = {{Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability}},
  year = {2026},
  eprint = {2602.01480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.01480},
}

Loading related content...

Figure