Some efficient language models replace attention with a fixed-size memory matrix that is updated once per token by the delta rule: read what the memory currently predicts for a key, then write back a correction proportional to the error. These models recall facts far better than earlier recurrent designs. Over very long inputs, however, they fail badly. SANE reports that a 0.4B delta-rule model (RWKV-7) has its memory norm grow without bound and produces NaNs around 20 million tokens. Its fix is to squash the memory with a tuned $\tanh$ every 16 tokens, and that fix works only inside a narrow threshold window. We start from a simple observation. In exact arithmetic, the pure delta rule is a relaxed projection: each row of the memory is moved toward the hyperplane defined by the current key and value. A projection of this kind cannot grow the memory faster than the square root of the sequence length. Exponential blow-up therefore has to come from one of two places. Either the model uses transitions outside this safe family, or rounding in the fast parallel kernel breaks the projection property.
The proposal has two parts. First, we diagnose which of these two causes is responsible, by replaying identical 20-million-token streams through bf16 chunked kernels and through fp64 sequential recurrences. Second, we build a drop-in replacement for SANE whose boundedness is proved rather than tuned. Each chunk's update is computed so that the result is the exact update for slightly perturbed keys; this property is called backward stability. A nonexpansive radius cap is then applied only at event boundaries, which are the points where the model's own next-token surprise spikes. The quantity we care about is functional health over 100M tokens, and the test is associative recall, not merely the absence of NaNs.
Core Claim
Write the memory as $S_t \in \mathbb{R}^{d_v \times d_k}$, with unit-norm key $k_t$, value $v_t$ and write strength $\beta_t$. The delta update $S_t = S_{t-1} + \beta_t (v_t - S_{t-1}k_t)k_t^T$ moves each row of $S$ by a relaxed Kaczmarz step toward the hyperplane $\{s : s^T k_t = v_{t,i}\}$. A two-line calculation (maximize the change in squared row norm over the current prediction $s^T k_t$) gives
$$\|S_t\|_F^2 \;\le\; \|S_{t-1}\|_F^2 + \frac{\beta_t}{2-\beta_t}\,\|v_t\|^2 .$$
In exact arithmetic, then, the memory grows at most like $\sqrt{t}$ for $\beta_t \in [0,1]$. At 20M tokens that is a factor of a few thousand, nowhere near bf16 overflow. The constant diverges as $\beta_t \to 2$, which is the reflection regime that DeltaProduct uses to obtain negative eigenvalues for state tracking.
Our claim has three parts. Exponential runaway of the kind SANE reports is not intrinsic to error-correcting memory writes. It can be removed by computing the parallel chunk update in a backward-stable way and adding one projection onto a norm ball, an operation that can only shrink distances. With both in place, the bound above holds for the computed memory up to a rounding term, with no threshold window to tune. The new empirical question is where that projection should be applied. We hypothesize that applying it at surprise-detected event boundaries keeps within-event associations intact, and so preserves recall that uniform 16-token squashing destroys. That hypothesis could fail, and the experiments are designed to show it if it does.
How the Four Directions Fit
The delta rule is the mechanism, and associative memory capacity, meaning how many key-value pairs a $d_k$-dimensional memory can hold and recall, is what we measure. DeltaNet showed that this update is a generalized Householder transformation, a rank-one reflection-like operator. It made training fast by grouping tokens into chunks and expressing each chunk's product of such operators in the compact WY form from the incremental orthogonalization literature. That form requires inverting a small unit lower-triangular matrix built from the chunk's key correlations.
This inversion is selective inversion: an explicit inverse of a small diagonal block, used in place of a long serial substitution. It is the same move made by communication-avoiding triangular solvers. Its accuracy depends on how well-conditioned the block is, and work on quantized Gated DeltaNet already finds that truncation errors in these inverses propagate into later updates. That makes it a plausible source of the numerical failure mode.
The unconditional backward stability literature, including Householder sketching for least squares and backward error analysis of reorthogonalized Lanczos, supplies the target property: the computed result is the exact answer for a nearby input. We reuse its analysis style rather than its algorithms.
Prediction error boundary detection, the practice of marking an event boundary wherever a predictor is suddenly surprised, decides where the cap goes. We borrow the online discriminator from ESTimator, which compares each new error against a running Gaussian fit of recent errors. We apply it to next-token negative log-likelihood, the surprise signal EM-LLM uses to segment its cache.
One clarification matters for the novelty claim. Moving the parallel chunk boundaries to event boundaries changes nothing in exact arithmetic, because the chunked algorithm computes the same recurrence for any chunking. We therefore do not propose ragged, event-sized kernels. Events determine only where the nonexpansive cap acts and where we spend fp32 effort on re-stabilizing the memory.
Research Program
The first stage is diagnosis. We take RWKV-7 0.4B, as used by SANE, together with DeltaNet, Gated DeltaNet and DeltaProduct models of matched size that we train on SlimPajama. For each, we replay the same long streams under three settings: the standard bf16 chunked kernel, the same kernel with fp32 triangular inversion and fp32 memory accumulation, and an fp64 sequential recurrence. In all three, the rest of the network stays in bf16. Along each stream we log three quantities: the memory's Frobenius and spectral norms, the condition number of each chunk's triangular block, and the gap between the computed chunk transition and the nearest exact product of Householder-type operators. If the fp64 run also explodes, the cause is algebraic and lies in transitions outside the safe family. If it does not, the cause is rounding.
We also test a specific prediction. Within an event, keys are highly similar; Hierarchical Episodic Memory reports up to 2.04x higher within-event key modularity than fixed segmentation. We therefore expect intra-event chunks to produce the worst-conditioned triangular blocks, so that rounding damage should concentrate inside long stationary stretches.
The second stage is construction, and the chosen fix depends on the diagnosis. If rounding is the cause, each chunk is computed in a backward-stable way. Keys are renormalized in fp32, and the B×B triangular system is solved by substitution in fp32 rather than by a bf16 explicit inverse. This costs little at B=64 and leaves the large matrix products on tensor cores. We then prove that the computed memory satisfies the growth bound above plus a term of order machine precision per write. If the transition family is the cause, we constrain $\beta_t \le 1$ for value-writing steps and keep reflections ($\beta \to 2$) only in a separate state-tracking factor. The bound shows why this separation is necessary.
In both cases we add one operation: whenever the online surprise discriminator fires and $\|S\|_F$ exceeds a radius $R$, the memory is rescaled onto the ball of radius $R$. This projection onto a convex set can only shrink distances, so it never amplifies error. $R$ is set once from the observed $\sqrt{t}$ envelope and is not tuned per task.
The third stage measures capacity. On multi-query associative recall (MQAR) from Zoology, we sweep the number of stored pairs relative to $d_k$ and the length of the distractor gap out to millions of tokens. This lets us check whether stabilization trades boundedness for recall, which is the "numerically healthy but functionally dead" failure SANE documents.
Baselines
The naive reference is the unmodified bf16 chunked kernel of each model, which by SANE's report collapses for RWKV-7 at roughly 20M tokens with 0.00 math accuracy afterwards.
The strongest existing method is SANE itself, the input-dependent $\tanh$ compression every 16 tokens. SANE reports an average accuracy of 35.56 across GSM8k, MATH-500 and ASDiv after a 100M-token prefix at $\alpha=3$ on RWKV-7 0.4B. It is the only method in the file evaluated at this scale of context with a functional metric. We run SANE's released settings at $\alpha \in \{3,4,5\}$ and our method on identical streams, base weights and evaluation prompts, with both applied training-free at inference. Beating SANE means matching or exceeding 35.56 at 100M tokens with a single untuned $R$, and degrading less on short-context accuracy, where SANE's aggressive settings hurt.
Precision alone is a baseline: fp32 accumulation and inversion with no cap. It isolates how much of the benefit comes simply from higher precision.
A decay baseline is Gated DeltaNet's forgetting gate, which bounds the norm by shrinking all memories uniformly.
The key ablation applies the identical radius projection at three placements with the same total count: fixed 16-token intervals, random positions, and surprise boundaries. Any gain from boundary gating must appear in this comparison.
The non-parametric reference point is EM-LLM, with 100% passkey accuracy at 10M tokens. It is a ceiling with unbounded memory, not a competitor at fixed state size.
For MQAR, DeltaNet is reported to reach "near-perfect accuracy even at low dimensions", but no exact number was found on Lacuna for DeltaNet on MQAR. We will report our own runs of all models under a shared training budget.
Why not just…?
Why not just run the recurrence in fp32? This is a real threat, and our first experiment tests it directly. If fp32 accumulation alone removes the blow-up and preserves recall, then the backward-stable chunk step reduces to a precision recommendation. In that case we would narrow the contribution to the diagnosis plus the boundary-gated cap, which fp32 alone does not supply: the exact-arithmetic $\sqrt{t}$ growth still pushes the memory far from the scale it was trained at. If fp32 instead only delays failure, as the conditioning hypothesis predicts for long stationary stretches, the structured fix is warranted.
Why not just tune SANE's threshold or use weight decay? SANE itself reports that ordinary L2 decay does not stop the runaway. It also reports that its own threshold is safe only for $3 \le \alpha \le 5$, below which short-context reasoning drops and above which collapse returns. Our cap is a projection that can only shrink distances, and its radius comes from a proved envelope. The comparison across SANE's full $\alpha$ range shows whether removing the tuning window costs any accuracy.
Why not just add a forgetting gate? A gate keeps the norm bounded by shrinking every stored association at every step, which caps how long a fact survives regardless of how important it is. The MQAR sweep over distractor length directly measures this: a decay model should lose recall as the gap grows even at low load, while a radius cap triggered only at boundaries should not.
Why not just interleave a few attention layers? Zoology shows that sparse attention closes 97.4% of the perplexity gap. At 100M tokens, however, even a few attention layers need a cache that grows linearly with context, and the regime we target is fixed-size memory. We treat hybrids as out of scope rather than beaten.
Decisive Checks
The diagnosis is decisive by construction. Either the fp64 sequential replay explodes or it does not, and the triangular condition numbers either track key similarity or they do not.
The method succeeds if, on SANE's english_math_chat and mixed_chat streams to 100M tokens, the backward-stable kernel with cap stays finite on every stream and matches or exceeds 35.56 average math accuracy with one untuned $R$. Throughput must stay within 15% of the stock chunked kernel. DeltaNet reports up to a 30x speedup from chunking, so a loss of that size would erase the method's reason to exist. State-tracking on $S_4$ and $S_5$ must be unchanged relative to DeltaProduct, confirming that separating write strengths does not cost expressivity.
The boundary hypothesis is supported only if surprise placement beats count-matched random and fixed placement on MQAR recall at high load and on PG-19 long-range perplexity. We also check boundary quality against the human-annotated podcast transcripts used by Hierarchical EM. If placement makes no difference, we will report that the stability result stands alone and that event boundaries matter for memory organization but not for memory health.
References
- SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length
- A parallel algorithm for incremental orthogonalization based on the compact WY representation
- Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations
- When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet
- Householder Sketch for Accurate and Accelerated Least-Mean-Squares Solvers
- Backward error analysis of the Lanczos bidiagonalization with reorthogonalization
- Online Generic Event Boundary Detection
- Human-inspired Episodic Memory for Infinite Context LLMs
- Hierarchical Episodic Memory in LLMs via Multi-Scale Event Organization
- Zoology: Measuring and Improving Recall in Efficient Language Models