Symmetry Breaking at the Edge of Stability via Central Flow Analysis

Naima Elosegui Borras, Georgios ArvanitidisarXiv 2026 · arXiv

TL;DR

This paper investigates how the "Edge of Stability" (EOS)—a regime where neural network training oscillates due to high learning rates—breaks the architectural conservation laws (like layer balancedness) typically found in smooth gradient flow. By applying central flow analysis, the authors derive a mathematical "drift" term that predicts exactly which symmetries will fail and how the network's internal invariants will evolve during these oscillations.

What Problem Does This Paper Address?

In ideal, continuous-time gradient flow, neural networks often possess "conserved quantities." For example, in a simple two-layer linear network $uv$, the difference between the squared weights of the layers ($u^2 - v^2$) remains constant throughout training. This is known as balancedness.

However, real-world training uses discrete Gradient Descent (GD) with a finite learning rate $\eta$. When $\eta$ is large, the optimizer often hits the Edge of Stability (EOS), where the loss doesn't diverge but instead oscillates wildly around a trajectory. Previous research showed that these finite steps "break" the conservation laws, but we lacked a precise way to calculate this breaking. This paper seeks to bridge that gap: if we know the geometry of the network and the nature of the EOS oscillations, can we predict how balancedness will drift?

Methods

The authors utilize the Central Flow framework (Cohen et al., 2025), which describes the time-averaged trajectory of a system undergoing EOS oscillations. The central flow adds a "curvature correction" to the standard gradient:

$$\dot{\theta} = -\eta\nabla L(\theta) - \frac{\eta}{2} \nabla_\theta \langle H(\theta), \Sigma \rangle$$

Here, $H$ is the Hessian (curvature) and $\Sigma$ is the covariance of the fast oscillations. The core contribution of this paper is deriving the symmetry-breaking drift matrix $\tilde{\mathcal{B}}$.

For three specific architectures, they calculate how this drift interacts with known conservation laws:

  1. Scalar Factorization: A toy model ($uv$) to isolate the mechanism.
  2. Matrix Factorization: A linear model ($UV^\top$) where invariants are matrix-valued.
  3. Two-layer ReLU Networks: A non-linear model where conservation laws are neuron-wise.

The authors define a survival criterion: a conservation law survives the EOS if its direction is orthogonal to the span of the breaking matrices $B(t)$ generated by the oscillations.

Table 1: The three instances of the symmetry-breaking construction. Here $\Psi = U^\top U - V^\top V$. The raw matrix $\tilde{\mathcal{B}}$ is projected onto the architecture-dependent admissible test space before determining survival.

Datasets

The study uses synthetic datasets to verify the theoretical derivations:

  • Scalar/Matrix Factorization: Trained to reconstruct a target value or a rank-2 target matrix $Y$.
  • ReLU Networks: Trained on a synthetic regression task where inputs $x_i$ and teacher weights $w_\star$ are drawn from standard normal distributions $\mathcal{N}(0, I)$.

Key Insights and Ideas

  • Geometric Breaking: Symmetry breaking isn't random; it is a deterministic drift caused by the interaction between the oscillation covariance and the third-order derivatives (the gradient of the Hessian) of the loss function.
  • The "Balancedness Gap" Shrinks: In the scalar case, the authors prove that if the network is unbalanced ($h \neq 0$), the EOS oscillations produce a drift that pushes the factors toward balance ($hh \leq 0$).
  • Covariance Matters: Survival of a law doesn't just depend on the weights being balanced; it requires "covariance balance." Even if $U^\top U = V^\top V$, a law can still break if the oscillations in $U$ are different from the oscillations in $V$.
  • ReLU Boundary Sensitivity: For ReLU networks, the conservation laws are only "certified" as long as the oscillations stay within the same activation pattern (i.e., the same neurons stay "on" or "off"). If the "bounce" is big enough to flip a ReLU, the Hessian changes discontinuously, and the standard central flow approximation becomes a heuristic.

Key Results

  • High-Precision Validation: In simulations, the authors measured the actual drift of conserved quantities and compared it to their analytic predictions. The discrepancy was remarkably low (relative errors of $10^{-9}$ to $10^{-11}$), confirming that the central flow accurately captures symmetry breaking.
  • Completeness of Breaking: In the matrix factorization experiments (rank $r=2$), they found that the breaking direction eventually spans the entire space of possible invariants. This means that at the EOS, no constant conservation laws survive for the entire trajectory.
  • Saturation of Bounds: For the scalar model, the authors derived a stability bound for the long-time value of the balancedness gap. Empirical runs showed that the network actually saturates this bound, "freezing" at a specific non-zero unbalanced state once oscillations terminate.

Limitations and Future Work

  • Activation Boundaries: The current ReLU analysis assumes a fixed activation pattern. The authors note that characterizing drift when oscillations cross ReLU boundaries is a significant open challenge.
  • Counting Laws: While the paper identifies how laws break, it does not yet provide a general formula for the number of surviving laws in arbitrary deep architectures.
  • Trajectory-Dependent Laws: The authors suggest that while old laws break, new "trajectory-dependent" laws might emerge at the EOS, which remains to be explored.

Summary updated

Abstract

Continuous symmetries in neural networks lead to conserved quantities that are invariant under gradient flow, such as layer balancedness in linear and ReLU networks. Finite-step gradient descent need not preserve these laws, particularly at the edge of stability (EOS), where the iterates oscillate around a smoother trajectory, which can be described by the central flow dynamics (Cohen et al., 2025). We study how gradient flow conservation laws evolve under the central flow description of this regime. We show that the curvature correction induces a finite-dimensional drift determined by the oscillation covariance along critical parameter directions. We thus take the first steps to analyse which conservation laws remain and are broken at the EOS according to the central flow dynamics by analysing a scalar two-layer model, two-layer matrix factorisation, and a two-layer ReLU network, where the known conservation laws are shown to be complete (Marcotte et al., 2023).
BibTeX
@misc{borras2026symmetry,
  author = {Naima Elosegui Borras and Georgios Arvanitidis},
  title = {{Symmetry Breaking at the Edge of Stability via Central Flow Analysis}},
  year = {2026},
  eprint = {2611.06718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06718},
}
Figure