Rectified Looped Transformer

Musheng Li, Jiawei Zhang, Ziyuan Liu, Yuantao Gu

TL;DR

The Rectified Looped Transformer (RLT) stabilizes parameter-efficient looped models by replacing direct feedback with a flow-inspired interpolation schedule. By treating each loop's output as an endpoint prediction and moving toward it linearly, RLT prevents hidden-state "blow-up" and reduces error amplification, leading to better language modeling and significantly greater robustness to changes in inference depth.

What Problem Does This Paper Address?

Looped Transformers are a clever way to scale model depth without exploding the parameter count. Instead of a massive stack of unique layers, you repeatedly pass data through the same shared block. However, this recurrence introduces a stability problem.

In a Standard Looped Transformer (SLT), the output of one loop becomes the direct input for the next. As the model "thinks" longer (more loops), two things often happen:

  1. Scale Growth: The magnitude (norm) of the hidden states can grow uncontrollably.
  2. Perturbation Amplification: Small numerical errors or input noise can be amplified exponentially as they pass through the shared block repeatedly.

These issues make SLTs fragile, especially when you try to use a different number of loops at inference time than you did during training. The authors of this paper set out to find a feedback mechanism that makes these models more stable and reliable.

Methods

The core innovation is Rectified Looped Transformer (RLT). Instead of the "direct feedback" used in SLT, RLT uses a linear interpolation schedule inspired by rectified flow modeling.

At each loop $k$, the model has a current state $x_k$ and produces a prediction $h_{k+1}$. Instead of setting $x_{k+1} = h_{k+1}$, RLT calculates the next state as: $$x_{k+1} = (1 - \eta_k)x_k + \eta_k h_{k+1}$$ where the interpolation weight is defined as $\eta_k = \frac{1}{K - k}$ ($K$ being the total number of loops).

Figure 1: Overview of Rectified Looped Transformer (RLT). (a) SLT and RLT share the same looped architecture and uniform all-step supervision. RLT replaces direct feedback with interpolation between the current input and the predicted endpoint. (b) Schematic latent dynamics under matched and mismatched loop counts, illustrating more uniform state advancement and more stable final predictions in RLT, while SLT may fall short or drift beyond the target region.

This simple change has a profound geometric meaning: each loop treats the network output as a prediction of the final endpoint and moves the current state one step closer to that endpoint along a straight line. As $k$ approaches $K$, the weight $\eta_k$ increases to 1, ensuring the final loop's output is fully utilized.

Datasets

The authors evaluated RLT against SLT using:

  • Training: A deduplicated version of The Pile (6.88B tokens).
  • Language Modeling Evaluation: Perplexity (PPL) measured on The Pile, FineWeb-Edu (FWE), and OpenWebText (OWT).
  • Downstream Benchmarks: Ten tasks including HellaSwag, PIQA, ARC, WinoGrande, and LAMBADA to test reasoning and general knowledge.

Key Insights and Ideas

  • From Exponential to Polynomial: Theoretical analysis shows that while SLT's sensitivity to initial perturbations can grow exponentially with depth, RLT's sensitivity is bounded by a polynomial. This is a massive win for numerical stability.
  • Uniform State Magnitudes: RLT keeps hidden-state norms bounded across loops. In contrast, SLT's norms tend to accumulate, which can lead to saturation or numerical instability in deep recurrence.
  • Adaptive Step Sizes: Because the interpolation depends on the total number of loops $K$, RLT naturally adjusts its "step size" if you change the loop count at inference time. This allows the model to stay on a stable trajectory even if it's allowed more (or fewer) steps to "think" than it saw during training.
  • Zero-Cost Gains: Remarkably, these improvements require zero additional trainable parameters and no extra computational overhead per loop. It is purely a change in how the data is routed between iterations.

Key Results

  • Better Language Modeling: At matched training and inference depths (e.g., 4 loops), RLT reduced perplexity by 4.2% to 5.5% across different corpora compared to SLT.
  • Inference Depth Flexibility: RLT is much more robust to "mismatched" depths. When a model trained with 8 loops was tested with 32 loops, SLT's perplexity exploded to 57.72, while RLT remained stable at 17.72.
  • Robustness to Noise: When Gaussian noise was injected into the initial embeddings, RLT exhibited significantly smaller output displacements than SLT, confirming the theoretical "perturbation amplification" benefits.
  • Quantization Friendly: RLT maintains its performance advantage even under aggressive quantization (e.g., W4A16), likely due to its more stable internal activations.

Limitations and Future Work

  • Fixed Schedule: The current interpolation schedule is predefined based on the loop count. Future work could explore adaptive schedules where the model decides how much to advance based on the complexity of the input.
  • Scale of Evaluation: While 278M parameters is a standard research scale, testing RLT in multi-billion parameter "frontier" models would be necessary to confirm its benefits for large-scale LLMs.
  • Theoretical Assumptions: Some theoretical bounds rely on Lipschitz conditions that may be difficult to enforce strictly in practice, though the empirical results suggest the stability holds regardless.

Not discussed in the paper: The impact of RLT on training speed (wall-clock time) or its performance on extremely long-context tasks beyond 1024 tokens.

Summary updated

Abstract

Looped Transformers increase effective depth by repeatedly applying a shared block, offering a parameter-efficient approach to language modeling. However, most existing works directly reuse each loop's output as the next's input, which can lead to hidden-state scale growth and perturbation amplification as recurrence deepens. We propose Rectified Looped Transformer (RLT), which redesigns feedback between loops using an interpolation schedule inspired by flow-based modeling. At each loop, RLT forms the next input by linearly interpolating between the current state and its prediction. Such interpolation distributes state changes across loops, encouraging a smoother trajectory from the initial input to the final output. Theoretical analysis proves that, under appropriate assumptions, RLT keeps state magnitudes uniformly bounded across loops and reduces the worst-case perturbation amplification. We evaluate RLT on language modeling corpora and downstream benchmarks. At matched training and inference depths, RLT consistently achieves lower perplexity than a standard looped Transformer, while exhibiting greater robustness to input perturbations and variance in inference depth. Consistent with the theoretical results, latent trajectory analysis shows that RLT maintains more stable hidden-state norms across loops, which also facilitates extrapolation-based decoding strategies. Notably, RLT achieves these gains without requiring additional trainable parameters or extra recurrent block evaluations, thus providing a simple yet more stable feedback mechanism for looped Transformers.
BibTeX
@misc{li2026rectified,
  author = {Musheng Li and Jiawei Zhang and Ziyuan Liu and Yuantao Gu},
  title = {{Rectified Looped Transformer}},
  year = {2026},
  eprint = {2611.16265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16265},
}
Figure