A looped transformer applies the same block to its hidden state over and over, so the number of loops acts as a dial for how much computation each input gets. These models can usually be run for more loops at test time than they saw in training, but how many loops each token should get is still set by hand. Two things go wrong as a result. Easy tokens keep iterating long after they have settled, and some models drift and lose accuracy when pushed past their training depth. This proposal asks whether the model's own dynamics can say when a token is finished. The specific quantity is the local rate at which small perturbations to a token's hidden state grow or shrink from one loop to the next, a finite-time Lyapunov exponent. The hypothesis is that a token still doing useful work sits near the "edge of chaos," where perturbations neither die out nor blow up. A finished token contracts. A token that is about to be corrupted by extra loops starts expanding.
Prior work measured this exponent only as one offline number for an entire fixed-depth model and found that well-performing looped attention models sit near zero (Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians). We propose measuring it per token and per loop at inference time, cheaply, and testing two things. The first is whether it tracks the moving "computation frontier" that adaptive-depth models develop. The second is whether a two-sided rule (stop on contraction, roll back on expansion) gives a better accuracy-per-compute trade-off than simpler stopping signals. The comparison that matters most is against the simplest one: stop when the hidden state stops changing.
Core Claim
The claim is that the local Lyapunov exponent separates three states that cheaper signals blur together. A token can be still computing, which shows as an exponent near zero. It can be done, which shows as a clearly negative exponent. Or it can be degrading, which shows as a clearly positive exponent. The third state matters because it explains a known failure: additive looped transformers lose accuracy when run past their training depth, which motivated both Looped Transformers with Source-Centered State Evolution and LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. We predict that this degradation sets in when per-token exponents cross from near-zero into the positive range.
A hidden-state residual $\|h_t - h_{t-1}\|$ is large both for a token making progress and for a token wandering chaotically, so it cannot tell those apart. An output-confidence score can be high on a token that has collapsed early onto a wrong fixed point. If the exponent does carry this extra information, it supplies the missing link the field currently lacks. The file records that fixed repetition matches adaptive halting on parity and addition (Comparing Fixed and Adaptive Computation Time for Recurrent Neural Networks), yet fixed loops fail to length-generalize (Adaptive Recurrence as Algorithmic Time for Length Generalization in Addition). Our reading is that uniform depth works when the chosen depth stays inside every token's near-critical window and fails when it runs past it.
If the exponent carries nothing beyond the residual, we will say so. In that case the edge-of-chaos story reduces to ordinary fixed-point detection for this purpose.
The Signal
For token $i$ at loop $t$, let $J_t^{(i)}$ be the Jacobian of that token's hidden-state update with respect to its previous state. We track one tangent vector $v$, propagated with a forward-mode Jacobian-vector product and renormalized every loop. Over a window of $k$ loops, the finite-time exponent is
$$\lambda_t^{(i)} = \frac{1}{k}\sum_{s=t-k+1}^{t} \ln \frac{\|J_s^{(i)} v_{s-1}\|}{\|v_{s-1}\|},$$
where $v_{s-1}$ is the renormalized tangent from the previous loop. This is the standard tangent-propagation estimate of the maximum Lyapunov exponent defined in the file, truncated to a short window so it can change over the course of the computation.
The Jacobian-vector product costs roughly one extra forward pass of the block. It can be computed every few loops instead of every loop, and it can be replaced by a finite-difference estimate, which runs a perturbed copy of the state alongside the original in the same batch. All reported compute figures include this overhead.
The halting rule has two sides. A token stops once $\lambda_t^{(i)} < -\tau_c$ for $m$ consecutive loops. If $\lambda_t^{(i)} > \tau_d$, the token is reset to its state just before the crossing and frozen there. The thresholds $\tau_c$, $\tau_d$ and the patience $m$ are fit once per trained model on a validation split. They are then held fixed across task lengths and difficulty levels. This directly targets the per-model threshold sensitivity that Dynamic Early Exit in Reasoning Models reports for confidence-based exiting.
Research Program
The work has three stages. Each stage can fail on its own terms, and each failure would change the conclusion.
Stage 1: Does the exponent track computation? We train single-layer looped transformers with PonderNet halting on least-significant-digit-first decimal addition, reproducing the setup of Adaptive Recurrence as Algorithmic Time for Length Generalization in Addition. That paper found a diagonal "computation frontier": the $k$-th answer digit becomes decodable only after a proportional number of loops. Its evidence was observational, and it left the cause open. We compute per-token exponents over the loop-by-digit grid and overlay them on the decodability map. The prediction is that a band of near-zero exponent moves with the frontier, and that digits behind it are contractive.
We then run a causal check. We apply a small Jacobian spectral penalty during a short fine-tune that pushes frontier tokens toward contraction, then measure whether length extrapolation breaks. Recurrent Self-Attention Dynamics supplies the spectral-regularization method used here.
Stage 2: Does degradation past training depth correspond to expansion? We train additive looped models and source-centered models on WikiText-103, following the setups of Looped Transformers with Source-Centered State Evolution, and we take LoopFormer-style shortcut-modulated models on FineWeb-Edu. We run each well past its training depth. For each, we measure two loop counts: where perplexity starts to rise, and where the fraction of tokens with positive exponent starts to rise. If the two coincide in the additive models, and both are absent in the source-centered models (which are designed to be stable around an anchor), that is direct evidence for the dynamical account of over-iteration. It would also test the source-centered paper's open question of whether stability holds away from its anchor.
Stage 3: Is it a useful halting rule? On Sudoku and parity, which have heterogeneous difficulty, and on addition beyond the training length, we compare the exponent rule against the alternatives below. Each method is evaluated on a full accuracy-versus-compute curve, not at a single operating point.
Baselines
The naive reference point is uniform depth tuned on validation data: the Repeat-RNN recipe of Comparing Fixed and Adaptive Computation Time for Recurrent Neural Networks, applied to the same looped transformer. It is the right floor because that paper shows it matches adaptive halting on parity. It solved parity in 22k training steps against 53k for adaptive computation time. Any adaptive rule that cannot beat a well-tuned fixed depth at matched compute has no case. We sweep the depth over the same compute range the adaptive methods use.
The strongest learned competitor is PonderNet (PonderNet: Learning to Ponder), a learned per-step halting probability regularized toward a geometric prior. It is the mechanism behind the best length-generalization result in the file. On addition, the PonderNet-based looped model of Adaptive Recurrence as Algorithmic Time keeps above 99% exact-match accuracy at 1.3 to 1.8 times the training operand length, and the fixed-loop variant fails. We use the same trained network for both methods. PonderNet's own halting head is compared against our rule applied to that network's states, so architecture and training data are identical and only the stopping decision differs. Beating it means matching its above-99% accuracy at fewer average loops, or extending the length at which accuracy stays above 99% at equal loops.
On Sudoku, the reference numbers are the roughly 90% accuracy reported for fixed-depth looped attention models whose global exponent lies in [−0.1, 0.1] (Recurrent Self-Attention Dynamics), and SATNet's 98.3% board-wise accuracy (SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver). SATNet is a structurally different solver and serves only as a ceiling. Our target is to match the fixed-depth looped model's accuracy at substantially fewer loops on easy boards. Board difficulty is measured by the number of givens.
For WikiText-103 extrapolation, there is no reported result found on Lacuna for a specific SCSE perplexity value. The file says only that SCSE kept the lowest perplexity from 8 up to 48 loops, so we rerun it ourselves.
The key ablations isolate the ingredient. The first is a residual-norm rule that stops when $\|h_t - h_{t-1}\|$ falls below a threshold. The second is an output-entropy rule in the style of Dynamic Early Exit in Reasoning Models. The third is a one-sided version of our rule with the contraction stop and no rollback. Every rule gets the same validation budget for threshold tuning, roughly the same number of threshold settings tried, and compute that counts the diagnostic overhead.
Why not just…?
Why not just stop when the hidden state stops changing? This is the most serious threat. A contracting token has a shrinking residual, so on clean tasks the two signals may agree almost everywhere. The case for the exponent rests on two situations where they should disagree. One is a token whose state moves a lot without making progress, in the expanding regime. The other is a token that moves slowly but is still sensitive to its inputs, near-critical with a small residual. Stage 2's over-iteration runs are where the first situation should show up. The residual baseline in Stage 3 is a direct head-to-head comparison. If the residual rule's accuracy-compute curve lies within seed variance of ours on every benchmark, we will conclude that Lyapunov monitoring adds nothing for halting. The Stage 1 and Stage 2 analyses would then stand as explanation only.
Why not just use output confidence or a learned halting head? Confidence reads only the final token distribution. Adaptive Thinking: Large Language Models Know When to Think in Latent Space argues that token entropy often fails to capture true difficulty. DEER's authors report that their threshold must be retuned per model, and that some underconfident models fail to exit even when their answer is correct. A learned head such as PonderNet is a real competitor, and the evidence for it is strong on addition. Its limitation is that it only knows the depths it was trained on and has no built-in alarm for drift. The cross-length and past-training-depth runs test exactly this: whether our fixed thresholds transfer to longer inputs better than PonderNet's learned head does on the same network.
Why not just tune a uniform depth? On parity and addition within the training length, this may simply win, as Comparing Fixed and Adaptive Computation Time found. We include it at every compute level. Our prediction is narrower. Uniform depth should lose when difficulty varies widely across inputs, as on Sudoku boards with different numbers of givens, and when inputs exceed the training length. The predicted mechanism is that no single depth stays inside every token's near-critical window. If uniform depth matches us on Sudoku difficulty bins, that prediction fails.
Why not just make the loop provably stable so halting doesn't matter? Source-centered models and energy-constrained models try this. Forcing energy-function structure through symmetric weights degraded accuracy in Recurrent Self-Attention Dynamics. Stability also does not remove wasted loops on easy tokens. We run our rule on source-centered models as well, where we expect only the contraction half of the rule to be active.
Decisive Checks
Three results would settle the proposal.
The first is from Stage 1. A near-zero-exponent band should co-move with the computation frontier on addition, and pushing that band toward contraction should hurt length extrapolation.
The second is from Stage 2. In additive looped models, the loop at which perplexity starts to rise past training depth should match the loop at which positive-exponent tokens begin to appear, to within a few loops. In source-centered models, both effects should be absent.
The third is from Stage 3. The two-sided rule should lie strictly above both the residual-norm curve and the tuned uniform-depth curve on Sudoku and on beyond-training-length addition, with the diagnostic cost counted. It should match PonderNet's above-99% extrapolation accuracy at fewer loops.
If the first two hold and the third does not, the contribution is a dynamical explanation of when uniform depth fails, not a better halting rule. If none hold, the edge-of-chaos picture does not describe per-token computation in looped transformers.
References
- Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
- Looped Transformers with Source-Centered State Evolution
- LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
- Comparing Fixed and Adaptive Computation Time for Recurrent Neural Networks
- Adaptive Recurrence as Algorithmic Time for Length Generalization in Addition
- Dynamic Early Exit in Reasoning Models
- PonderNet: Learning to Ponder
- SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver
- Adaptive Thinking: Large Language Models Know When to Think in Latent Space