Limit Cycles in Machine Learning Optimization

20 concepts11 papers1 proposal

Standard optimization theory presents a clean, comforting narrative: an algorithm navigates a loss landscape, slows down as the gradient approaches zero, and eventually comes to rest at a stable local or global minimum. But in practice—especially within complex models, high learning rates, and large-scale training regimes—this stationary picture is an illusion.

Instead of freezing at a single point, model parameters frequently enter a perpetual, dynamic dance known as a stable limit cycle or periodic orbit. In these states, weights continuously circulate through a fixed sequence of configurations without ever settling down, even as the overall training loss remains perfectly flat. Understanding why these cycles occur requires looking at optimization not as a static minimization problem, but as a discrete or continuous dynamical system.

Why Orbits Form: The Role of Non-Conservative Forces

To understand why an optimizer orbits rather than lands, we must look at the mathematical forces driving the parameter updates. In classical optimization, we assume the update steps are conservative—meaning they always follow the direct gradient of a single, static energy potential, like a ball rolling down to the bottom of a well. However, real-world training algorithms often introduce non-conservative forces that break this assumption, introducing rotational components into the update field.

Consider the noise inherent to Stochastic Gradient Descent (SGD). As demonstrated by Pratik Chaudhari et al. in Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks, gradient noise in deep networks is highly asymmetric (non-isotropic). This lopsided noise generates non-conservative forces that effectively transform a straight descent into a rotational field. Instead of dropping directly into the bottom of a valley, the optimizer is continuously pushed sideways, forcing the weights to orbit critical points or saddle points in deterministic limit cycles even after the training process has reached a steady state.

A similar phenomenon emerges in distributed environments. In Iterated Vector Fields and Conservatism, with Applications to Federated Learning, J Keith Rush et al. showed that taking multiple local gradient steps before averaging—a core mechanism of federated learning—renders the global update field non-conservative. This manifestation of a rotational field prevents the algorithm from reaching a stable equilibrium, forcing the model parameters into periodic limit cycles that orbit the optimal region without ever converging.

Step Size as a Geometric Filter

Even in the absence of asymmetric noise, the discrete nature of optimization steps can induce cyclic behavior. When we choose a learning rate, we are doing much more than adjusting the speed of convergence; we are altering the stability of the underlying dynamical system.

As explored by Kamil Nar et al. in Step Size Matters in Deep Learning, if the step size is too large relative to the local curvature of the loss landscape, a fixed point that would be stable under infinitesimal steps becomes unstable. The optimizer overshoots the minimum, gets flung back, and eventually settles into a stable periodic orbit around it.

This is not an isolated anomaly. Alexandru Crăciun et al. proved in Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations that excessive step sizes naturally induce these stable periodic orbits across realistic architectures (like CNNs and Transformers) using non-smooth activations like ReLU. These oscillations are entirely predictable based on the local geometry of the loss function and the chosen step size, explaining a common failure mode where models fail to reach a stable equilibrium.

The Paradox of Flat Loss: An Ergodic Perspective

If the weights of a network are constantly moving through a loop, why does the training curve look so stable? Why don't we see the loss wildly oscillating?

The authors of Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective resolved this paradox by applying ergodic theory to optimization trajectories. Haochuan Li et al. showed that while individual weights never converge to a stationary point and maintain a non-zero gradient norm, their statistical distribution over time becomes invariant. Because the limit cycle confines the weights to a specific region of the parameter space where the average loss is uniform, the macroscopic training metrics appear perfectly stable, masking the microscopic trajectory circulating underneath.

A Ubiquitous Phenomenon Across Paradigms

Limit cycles are not unique to deep neural networks; they represent a fundamental characteristic of iterative update systems across machine learning:

  • Boosting: In the context of AdaBoost, Cynthia Rudin et al. in On the Dynamics of Boosting and Ingrid Daubechies et al. in The Dynamics of AdaBoost: Cyclic Behavior and Convergence of Margins demonstrated that when the weak learner is not perfectly optimal, the sample weights do not converge to a maximum-margin solution. Instead, the algorithm settles into stable periodic cycles, repeating a fixed sequence of example importances and resulting in suboptimal margins.
  • Pattern Selection: Even simple modifications to data presentation can trigger orbits. Paul W. Munro showed in Repeat Until Bored: A Pattern Selection Strategy that threshold-based pattern selection strategies can trap weights in non-optimal cyclic orbits, creating infinite loops distinct from traditional local minima.
  • Energy-Based and Biologically Inspired Models: In Restricted Boltzmann Machines trained via Contrastive Divergence, Tijmen Tieleman proved in On the Convergence Properties of Contrastive Divergence that the updates do not follow the true gradient of any objective function, allowing parameters to cycle indefinitely. Similarly, in Predictive Coding Networks, Simon Frieder et al. showed in (Non-)Convergence Results for Predictive Coding Networks that specific weight and step-size configurations cause node values to orbit unstable fixed points, breaking the common assumption that these networks always settle on a stable prediction.
  • Saddle-Point Problems: When alternating optimization schemes are applied to sparse quadratic programs where a true saddle point is absent, Robbie Vreugdenhil et al. noted in Principal Component Hierarchy for Sparse Quadratic Programs that the algorithm can enter a limit cycle, oscillating between candidate solutions rather than stabilizing.

By reframing optimization not as a static march toward a minimum, but as a trajectory within a continuous or discrete dynamical system, we gain a clearer understanding of why models behave the way they do. Limit cycles demonstrate that stability in machine learning does not require stagnation; often, a model successfully learning a task is simply an optimizer finding its rhythm in a perpetual orbit.

Key Foundational Work

Repeat Until Bored: A Pattern Selection Strategy (1991) — contributed 4 concept elements to this direction.
On the Dynamics of Boosting (2003) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stable_limit_cycles_d27bf988,
  title = {stable limit cycles},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stable-limit-cycles/rdr_d27bf988ad9b4e07860cee694c600c2f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_stable_limit_cycles_d27bf988_v1,
  title = {stable limit cycles},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stable-limit-cycles/rdr_d27bf988ad9b4e07860cee694c600c2f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake