Recurrent sequence models remember the past through a transition matrix that is applied once per step. If its eigenvalues have magnitude above one, the hidden state explodes. If they sit well below one, memory fades exponentially. Long memory therefore requires eigenvalues pressed right up against the unit circle, and there, training becomes fragile. The standard fix in linear state-space models (SSMs, recurrent layers whose state evolves linearly and can be computed in parallel) is a reparameterization: the network learns an unconstrained number $w$, and a fixed smooth function maps it to an eigenvalue that lies inside the stable region by construction. The best-understood example is StableSSM's derived map $f(w) = 1 - 1/(aw^2+b)$. It works, but its own authors report an unexplained cost: it sometimes ends at a worse final loss than unconstrained parameterizations that happen not to diverge.
A reparameterization does two separate jobs at once. It keeps eigenvalues inside the stable disk, which is feasibility. It also changes the effective step size near the boundary, which is geometry. Parameter projection, a classical tool from online learning and reinforcement learning that takes an ordinary gradient step and then snaps the parameter back to the nearest point of an allowed set, supplies feasibility with no geometry at all. That makes it the right instrument for pulling the two jobs apart. We propose a factorial study that trains SSMs with projection alone, with a boundary-scaled step size alone, with both together, and with the original reparameterizations. The aim is to determine which ingredient buys stability, which buys final accuracy, and whether decoupling them recovers StableSSM's stability without its loss penalty. A side benefit follows directly. Projection onto the full disk permits negative and complex eigenvalues at no extra cost, which positive-only reparameterizations cannot represent, so the same experiments test whether projection helps state tracking.
Core Claim
We claim that the stability benefit of reparameterizations like StableSSM comes mainly from their boundary geometry, not from their feasibility guarantee. We further claim that their final-loss shortfall comes from applying that same conservative geometry everywhere, including far from the boundary. If both claims hold, the following recipe should match StableSSM's stability at high learning rates while reaching lower final loss:
- Parameterize eigenvalues directly in Cartesian coordinates.
- Project them onto a disk of radius $\rho = 1-\epsilon$ after each step.
- Scale their step by a function of the distance to the boundary.
Concretely, for a diagonal eigenvalue $\lambda \in \mathbb{C}$ with loss gradient $g$ and base update direction $u(g)$ (plain gradient for SGD, the normalized direction for Adam), the update is
$$\lambda \leftarrow \Pi_{\rho}\big(\lambda - \eta\,(1-|\lambda|)^{p}\,u(g)\big),$$
where $\Pi_\rho$ rescales any $\lambda$ with $|\lambda|>\rho$ back to magnitude $\rho$, $\eta$ is the learning rate, and $p \ge 0$ controls how strongly steps shrink near the unit circle.
The exponent $p$ is not a free knob. A chain-rule calculation for StableSSM's map shows that near the boundary its derivative satisfies $f'(w) \propto (1-\lambda)^{3/2}$. This follows from $w \propto (1-\lambda)^{-1/2}$ when $\lambda \to 1$. Consequently, SGD on $w$ moves $\lambda$ by an amount proportional to $(1-\lambda)^{3}g$, while Adam on $w$, whose per-coordinate steps are roughly constant in size, moves it by roughly $(1-\lambda)^{3/2}$. Setting $p$ to these values gives a projected method that imitates "Best" to first order near the boundary. It differs from "Best" only in its behavior far from the boundary and in how feasibility is enforced. That is exactly the difference the experiments need to isolate.
Relation to Prior Work
Two papers motivate the question.
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story shows that as eigenvalues approach one, the hidden state becomes extremely sensitive to small parameter changes, carving narrow valleys in the loss landscape. It identifies eigenvalue reparameterization and diagonal structure with Adam as remedies, and flags learning the phase of complex eigenvalues as still hard.
- StableSSM derives the "Best" map by requiring a constant gradient-to-weight ratio. It shows this map is the only one that converged at learning rate 0.01 in a 3-billion-parameter language model, while exp and softplus diverged within 100 steps. It also notes the final-loss gap.
Neither paper separates feasibility from geometry, and StableSSM never tries iterate projection. We reuse their analysis to set $p$, and we test their implicit assumption that the smooth map is necessary.
Two papers supply the projection method and its guarantees.
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization combines projection with averaging of late iterates in convex problems.
- Breaking the Deadly Triad with a Target Network uses projections to keep off-policy linear reinforcement learning from diverging on Baird's counterexample.
Both show projection preventing divergence in settings where the danger is iterates leaving a safe set. Both are limited to linear or convex models, and neither addresses sensitivity inside the set, which is the recurrent problem. That limitation is the reason we pair projection with a geometry term rather than claiming projection alone suffices.
Two papers constrain the design.
- projUNN projects onto the unitary manifold, where every eigenvalue has magnitude one. It reaches 97.0% on permuted MNIST, but costs ResNet-9 ten points on CIFAR-10 (82% vs 92%). This warns that projecting onto a thin set is too restrictive, so we project onto the full disk's interior instead.
- Muon Optimizes Under Spectral Norm Constraints proves that Muon with weight decay $\lambda$ keeps the operator norm below $1/\lambda$. It evaluates only feedforward and attention matrices, never recurrent transitions. For dense transitions, this makes Muon both an existing implicit projection method and our main competitor.
Finally, Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues shows that extending eigenvalues from $[0,1]$ to $[-1,1]$ takes parity from 0.0 to 1.0 and modular arithmetic from 0.314 to 0.971 in DeltaNet. It achieves this through architectural gating, not projection. Our disk projection reaches the same spectral range as a byproduct, so it is a natural secondary test.
Research Program
Diagonal study. The core is a controlled factorial on diagonal complex SSMs of the S4D/LRU family. S4 and S4D-DFouT supply the architecture and initialization, with discrete-domain initialization from S4D-DFouT so that initialization is held fixed. The factors are:
- Feasibility: none, or projection onto radius $\rho$.
- Geometry: $p = 0$, $p$ matched to "Best" under the optimizer in use, and a schedule that uses the matched $p$ only when $1-|\lambda|$ falls below a threshold.
- Reference cells: the original exp, softplus, and "Best" reparameterizations.
We cross these with a learning-rate sweep that extends up to and beyond the 0.01 at which StableSSM observed divergence. We log the fraction of eigenvalues pinned at the projection boundary, the distribution of $|\lambda|$ over training, and the gradient-to-weight ratio StableSSM used as its design criterion. This lets us check whether the projected, geometry-matched method reproduces that ratio without the map.
Phase coordinates. Because Cartesian parameterization of $\lambda$ lets magnitude and phase be learned without polar coordinates, we separately track phase trajectories. The question is whether phase learning, flagged as hard by Zucchet and Orvieto, becomes easier or harder once the magnitude reparameterization is removed.
Dense extension. A second, smaller arm carries the question to dense transition matrices on state-tracking tasks. There, the spectral-norm ball $\|A\|_{op} \le \rho$ is the natural projection set. We implement it by clipping singular values, with a low-rank update in the style of projUNN when the gradient is low rank, and we compare it with Muon at weight decay $1/\rho$.
Baselines
Naive reference points. The first is direct parameterization with no projection, which StableSSM reports scoring only 5.6% on associative recall at 131,072 tokens. The second is the same model with global gradient-norm clipping and decoupled weight decay, the default a practitioner would reach for. Both share our architecture, data, step budget, and learning-rate sweep. Beating them is necessary but uninformative on its own.
Strongest diagonal competitor. This is StableSSM's "Best" reparameterization, because it is the only method in the file shown to survive high learning rates at scale. We run it from the authors' published form with the same model width, initialization, and optimizer. Its constants $a, b$ get the same tuning budget as our $\epsilon$ and threshold. Its reported LRA average is 86.48, against 86.09 for S4's exponential parameterization (StableSSM), and it reports 100% on associative recall at 131,072 tokens. Because our reruns will be smaller than the 3B setting, we compare against our own reproduction and treat the reported numbers as a sanity check.
Beating "Best" means one of two things. Either projected geometry-matched training reaches a statistically lower final loss or higher LRA accuracy across at least three seeds at matched learning rate, or it survives a strictly larger stable learning-rate range. Matching it while being simpler counts as a partial result, not a win. For long-context reference we also report S4's 88% on Path-X (S4) and S4D-DFouT's 87.89% on PathX-256 (Uncovering the Spectral Bias).
Strongest dense competitor. On dense transitions this is Muon, because it already enforces a hard spectral bound. We run it with identical matrices, weight decay set to give the same radius, and the same budget. On state tracking we additionally compare to the $[-1,1]$ DeltaNet results above, with parity at 1.0 and modular arithmetic at 0.971, from Unlocking State-Tracking.
Ablations. The key ablations are the off-diagonal cells of the factorial:
- Projection with $p=0$ asks whether feasibility alone is enough.
- Geometry without projection asks whether shrinking steps alone keeps eigenvalues inside the disk.
These ablations, not any single leaderboard number, carry the scientific claim.
Why not just…?
Why not just use StableSSM's "Best" map and move on? It is the real threat, and it may win. It is already stable at scale, and our geometry-matched projection is designed to imitate it near the boundary. The reason to run this study anyway is StableSSM's own observation that the map sometimes ends worse than unconstrained training, together with the fact that it confines eigenvalues to $(0,1)$ and so excludes the negative eigenvalues state tracking needs. If the thresholded-geometry cell shows no final-loss advantage and no state-tracking gain, we will report that the map is already near optimal and that projection is merely an equivalent implementation.
Why not just clamp the eigenvalue magnitude, which is what radial projection is? For diagonal models it is exactly that, and we do not claim a new operator. The contribution is the factorial result: whether clamping alone ($p=0$) suffices. Zucchet and Orvieto's analysis predicts it will not at high learning rates, because the sharp valleys lie inside the disk. The $p=0$ cell tests that prediction directly.
Why not just use weight decay and gradient clipping? The file gives two reasons to expect failure. Muon argues that AdamW with weight decay leaves individual singular values free to escape bounds. Clipping limits step size but not where the iterate ends up, and it does nothing about the ill-conditioned directions near the boundary. The naive baseline cell measures this rather than assuming it.
Why not just use Muon on the recurrent matrix? For dense transitions this is a strong, cheap alternative. Its implicit constraint is exactly a spectral-norm ball, so our explicit projection has to beat it on stability range or state-tracking accuracy to matter. If Muon matches us, that is itself a new finding, since Muon has not been evaluated on recurrent transitions, and we would recommend it.
Proposed Experiments
Synthetic probes. Following Zucchet and Orvieto, we train a linear student to match a teacher recurrence with eigenvalues at controlled distances from the unit circle, from $1-10^{-1}$ to $1-10^{-4}$. We then map each factorial cell's success region in the plane of learning rate versus memory length. This is the cleanest view of feasibility versus geometry.
Associative recall. We scale associative recall from 8K up to the 131,072-token setting StableSSM used. The decisive check is whether projected direct parameterization recovers from 5.6% toward 100%, and whether it does so with $p=0$ or only with matched $p$.
Long Range Arena. We run the full Long Range Arena suite at standard settings for final-accuracy comparisons against our "Best" reproduction.
Language modeling. We do a learning-rate sweep on WikiText-103 with a model of roughly 125M parameters. There is no reported result on Lacuna for "Best" at this scale, so our reproduction is the reference. The question is whether the divergence pattern StableSSM saw at 3B already appears there.
State tracking. Parity and modular arithmetic, with diagonal and dense variants, test whether disk projection's access to negative eigenvalues closes the gap that positive-only maps leave.
Decisive outcome. The proposal succeeds if two things hold. First, the projection-plus-geometry cell matches "Best" in stable learning-rate range. Second, the thresholded variant beats it on final loss or state-tracking accuracy across seeds, while projection alone fails at high learning rate. That pattern would establish that geometry buys stability and that the remaining loss gap is the price of applying it globally. Other patterns are informative in their own right:
- If projection alone matches "Best", reparameterization in these models is mostly about feasibility, contrary to current understanding.
- If nothing beats "Best", we will have shown it is close to the right design and why.
References
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story
- StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Breaking the Deadly Triad with a Target Network
- projUNN: efficient method for training deep networks with unitary matrices
- Muon Optimizes Under Spectral Norm Constraints
- Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
- Efficiently Modeling Long Sequences with Structured State Spaces
- Uncovering the Spectral Bias in Diagonal State Space Models
- Long Range Arena : A Benchmark for Efficient Transformers