Recurrent networks train badly over long sequences because the backward error signal is multiplied by one step-to-step Jacobian per time step. If those products grow, gradients explode. If they shrink, gradients vanish. The standard fix is to constrain the recurrent weight matrix itself: make it unitary, or cap its largest singular value. We argue that this constrains the wrong object. The rate at which gradients grow or decay is set by the network's Lyapunov exponents, which measure how fast nearby hidden-state trajectories separate or converge along the trajectories the network actually runs on its data. Those exponents depend on the weights, the nonlinearity's slopes, and the input drive together. A unitary matrix inside a tanh network is still contractive, and a matrix with norm above one can be perfectly stable once the nonlinearity saturates.
We propose to make the training iterates themselves satisfy a dynamical constraint. After each optimizer step, the recurrent parameters are projected back onto the set where the largest Lyapunov exponent, estimated on the current batch, sits inside a chosen band around zero. Projection here means moving the parameters to the nearest point satisfying the constraint, the move that keeps classical online and reinforcement learning stable. This brings the parameter-projection toolkit to the quantity that recurrent-dynamics theory identifies as governing gradient stability. It also yields a training-time control for chaotic data that, unlike teacher forcing, needs no ground-truth latent states.
Core Claim
Hold the architecture, data and compute fixed. A hard projection onto a band of the trajectory's maximal Lyapunov exponent should give better long-horizon learning than three alternatives: unitary or orthogonal projection, singular-value clamping of the weight matrix, and a soft Lyapunov penalty. The improvement should be clearest in nonlinear networks, where matrix spectra and trajectory exponents diverge.
For chaotic targets, a ceiling on the exponent that is annealed upward toward the data's own exponent should remove most loss jumps without access to latent ground truth. We expect it to fall short of state-space teacher forcing when the observation-to-latent inversion is easy. We treat that last point as a test, not a promise.
Where This Sits
The motivating evidence comes from dynamical analyses of recurrent networks. On Lyapunov Exponents for RNNs computes the full Lyapunov spectrum of input-driven networks by periodic QR re-orthonormalization at about 4% training overhead. It finds that character-level validation loss drops as the largest exponent approaches zero from below, and that on motion capture the mean exponent predicts performance better than the largest one. It measures these quantities but never enforces them; we enforce them.
Bifurcations and loss jumps in RNN training proves for piecewise-linear networks that two kinds of qualitative change in the dynamics cause the sudden loss spikes:
- Degenerate transcritical bifurcations occur when a stable state collides with an unstable one. They make gradients explode.
- Border-collision bifurcations occur when a fixed point crosses an activation boundary. They make gradients vanish.
The authors suggest that parameter interventions could steer training away from these events but do not build one. We use their cycle-finding tool, DurstewitzLab/SCYFI, as an independent auditor of whether our projection avoids them.
Generalized Teacher Forcing for Learning Chaotic Dynamics proves a constraint any parameter-space method must respect: modelling chaos forces exploding gradients. Its fix mixes predicted latent states with inferred true states, which shrinks the Jacobian product. That requires inverting observations into latents, and our method avoids this requirement. The same result also rules out a naive version of our idea. A fixed ceiling below the target's own exponent must make chaos unlearnable, which is why the chaotic-data variant anneals the ceiling.
The method lineage comes from projection in online learning. TD(lambda) Converges with Probability 1 and On the Convergence of SARSA with Linear Function Approximation keep iterates in a bounded box or ball so that noise cannot compound. Breaking the Deadly Triad with a Target Network shows on Baird's counterexample that projection gives stability even with the regularization weight set to zero, while unprojected TD errors reach $10^5$. That is our main motivation for preferring a hard constraint over a penalty.
These guarantees rely on convex sets, where projection never moves an iterate further from the optimum. Our constraint set is not convex, so we inherit the mechanism but not the proofs. We say this up front.
The closest existing work constrains matrix spectra:
- projUNN projects onto unitary matrices via cheap rank-one updates.
- scoRNN parameterizes orthogonal matrices through a scaled Cayley transform.
- Muon Optimizes Under Spectral Norm Constraints shows the Muon optimizer implicitly projects onto a spectral-norm ball, though only on feedforward and transformer models.
All three act on singular values of $W$ alone. The unitary methods also pin every singular value to one, which the projUNN authors pay for with a drop from 92% to 82% on CIFAR-10 ResNet-9. Our constraint is on a trajectory quantity and leaves the singular-value distribution free.
The Projection Operator
Let $\theta$ be the recurrent parameters and let $\hat\lambda(\theta)$ be the largest Lyapunov exponent. We estimate it by QR iteration along hidden-state trajectories of the current minibatch over a window of $T_\lambda$ steps. The feasible set is
$$\Theta_{[a,b]} = \{\theta : a \le \hat\lambda(\theta) \le b\},$$
with band edges $a<0<b$ near zero for memory tasks.
The exact projection onto this non-convex set is intractable. We therefore project onto its local linearization. Whenever a step leaves the band, for example by $\hat\lambda > b$, we apply
$$\theta \leftarrow \theta - \frac{\hat\lambda(\theta) - b}{\|\nabla_\theta \hat\lambda(\theta)\|^2}\,\nabla_\theta \hat\lambda(\theta),$$
which is the exact Euclidean projection onto the linearized half-space. We repeat at most $k$ times, and symmetrically for $\hat\lambda < a$. The gradient $\nabla_\theta \hat\lambda$ comes from automatic differentiation through the QR recursion over a short window.
To keep overhead near the reported 4% for spectrum estimation, the check runs every $K$ optimizer steps. We report wall-clock cost explicitly.
For motion capture, following the finding that the mean exponent is the better predictor, the band is placed on the mean exponent instead. For chaotic targets, $b$ starts near zero and is raised on a schedule toward $\lambda_{\text{data}}$. That target exponent is estimated from the observed time series by standard delay-embedding methods, so no latent inversion is needed. The BPTT window is chosen so that $b\,T$ stays bounded.
Research Program
The work has three parts.
Analysis. We characterize, for tanh and piecewise-linear recurrent networks, how far the band on $\hat\lambda$ and the singular values of $W$ come apart during training. The project's premise is that they decouple: networks with $\|W\|>1$ and $\hat\lambda<0$, and unitary networks with $\hat\lambda \ll 0$. If this gap turns out small in practice, singular-value clamping suffices and the project's premise fails early and cheaply.
Long-memory tasks. We test the projection where marginal stability is the right target.
Chaotic reconstruction. We test whether the annealed ceiling can substitute for state-space forcing. Throughout, SCYFI is run on piecewise-linear models at every logged loss jump to classify which bifurcation caused it. This tests directly whether the projection prevents degenerate transcritical events and whether it introduces new border-collision stalls.
Baselines
Unconstrained training with gradient clipping. This is the do-nothing reference point and the practitioner's default. We use the same Adam optimizer and the same width, with the clipping threshold tuned over the same grid size as our band edges. Beating it means higher task accuracy plus fewer loss jumps, counted as validation-loss increases above a fixed threshold.
projUNN. This is the strongest existing projection method on the long-memory benchmarks. On permuted MNIST its authors report up to 97.0% test accuracy, against 96.6% for scoRNN, 92.9% for LSTM and 94.7% for a full-capacity unitary network (projUNN, scoRNN). scoRNN reports solving the copy task at delay 2000, where LSTMs fall to the trivial baseline. We rerun projUNN and scoRNN in our codebase at matched hidden size, epochs and tuning trials, rather than relying only on reported numbers. Beating them means matching their copy-task success and exceeding 97.0% on permuted MNIST, or matching it on tasks where unitarity is a handicap. Beating them on CharRNN validation loss and motion capture would also count; for these no reported result was found on Lacuna, so our reruns set the reference.
Singular-value band clamping. This is the key ablation. After each step we compute the SVD of $W$ and clip its singular values into $[\sigma_{\min},\sigma_{\max}]$, tuned with the same budget. It also stands in for Muon with weight decay, which bounds only the top singular value; for Muon on recurrent networks no reported result was found on Lacuna, so Muon is run directly as a further baseline. This comparison isolates the core ingredient: constraining the trajectory exponent rather than the matrix.
Soft Lyapunov penalty. This adds $\beta\,\max(0,\hat\lambda-b)^2$ to the loss, with $\beta$ tuned over the same number of trials as our band edges. It isolates projection versus penalty.
Generalized teacher forcing. On chaotic data we use the authors' shallow piecewise-linear pipeline, DurstewitzLab/GTF-shPLRNN, with its metrics. These are state-space divergence $D_{stsp}$, a measure of how well attractor geometry is matched, and $D_H$, a measure of temporal power-spectrum error. The paper reports roughly an order-of-magnitude advantage over LSTMs, reservoir computing, SINDy and neural ODEs on EEG and ECG, but exact table values are not in our file. We will regenerate them from the released scripts. Beating it is not expected where linear inversion is available. Success there means closing most of the gap to it while beating unforced training, and outperforming it when inversion is deliberately made harder.
Why not just…?
Why not just clip gradients? Clipping limits the size of a step but not where the step lands. Bifurcations and loss jumps in RNN training argues that clipped training still crosses degenerate transcritical bifurcations and does nothing about border-collision vanishing. Our SCYFI audit of every loss jump under clipping versus projection tests this claim directly on our models.
Why not just clamp the singular values of $W$? This is the most serious threat. It is cheap, convex, and inherits all classical projection guarantees. Our argument is that in nonlinear networks the matrix spectrum and the trajectory exponent decouple, so clamping is either too loose to prevent explosion or too tight to keep memory. The analysis stage measures this decoupling before anything else is built. If clamping matches our method at equal tuning budget on CharRNN and the copy task, we will report that the cheaper constraint suffices and drop the claim.
Why not just add a Lyapunov penalty to the loss? A penalty trades task loss against stability through a weight that must be tuned and that permits transient excursions. Those excursions are exactly where bifurcations happen. The deadly-triad results show projection stabilizing where regularization alone did not (Breaking the Deadly Triad with a Target Network), but that evidence is from linear reinforcement learning. The matched-budget penalty ablation decides the question for recurrent networks.
Why not just use teacher forcing on chaotic data? Where latents are recoverable by a linear map, it will probably remain best, and it has a proof behind it (Generalized Teacher Forcing for Learning Chaotic Dynamics). Our method's case rests on settings where inversion is costly. We construct such settings by observing Lorenz-96 through a fixed random nonlinear map, and we use high-channel EEG with a reduced latent dimension. We report both regimes honestly.
Proposed Experiments and Decisive Checks
The long-memory suite covers the copy and adding tasks at delays 500, 1000 and 2000, permuted MNIST, CharRNN on War and Peace, and CMU motion capture. It uses both tanh and piecewise-linear recurrent networks at hidden sizes from 64 to 512, with five seeds. The chaotic suite uses Lorenz-63, Lorenz-96, EEG and ECG through the released pipeline. Alongside task metrics we log:
- gradient-norm traces;
- the estimated exponent, against the band;
- loss-jump counts;
- SCYFI bifurcation labels for piecewise-linear runs;
- wall-clock overhead.
Four results decide the project:
- Trajectory versus matrix constraint. Lyapunov-band projection must beat singular-value band clamping at matched compute on at least the nonlinear CharRNN and copy settings. Otherwise the core claim fails.
- Projection versus penalty. Hard projection must beat the soft penalty at equal tuning budget.
- Bifurcation avoidance. Loss jumps under projection should drop, with SCYFI attributing the remaining ones mostly to border-collision rather than degenerate transcritical events. If degenerate transcritical events persist, the projection is not steering away from the manifolds it targets.
- Necessity of annealing. A fixed ceiling below the data's exponent should fail to reconstruct Lorenz attractors, as the teacher-forcing theory predicts, while the annealed ceiling should succeed. Failure of this negative control would mean our exponent estimates or projections are not doing what we think.
References
- On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
- Bifurcations and loss jumps in RNN training
- Generalized Teacher Forcing for Learning Chaotic Dynamics
- TD(lambda) Converges with Probability 1
- On the Convergence of SARSA with Linear Function Approximation
- Breaking the Deadly Triad with a Target Network
- projUNN: efficient method for training deep networks with unitary matrices
- Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
- Muon Optimizes Under Spectral Norm Constraints