Architectures and KV Replay for Depth-Recurrent Language Models

Lucia Licakova, Niklas Schmidinger, Lukas Hauzenberger, Sebastian Böck, Sepp HochreiterarXiv 2026 · arXiv

TL;DR

Depth-recurrent language models save parameters by looping through layers, but they usually explode inference memory because every loop iteration requires its own Key-Value (KV) cache. This paper introduces Recurrent KV Replay, a training technique that allows multiple recurrence iterations to share the same KV states, alongside new hybrid architectures that interleave attention with fixed-size recurrent mixers to slash memory costs without sacrificing performance.

What Problem Does This Paper Address?

Modern Transformers are parameter-hungry. One way to make them more efficient is depth recurrence: instead of having 96 unique layers, you might have 24 layers and loop through them 4 times. This gives you the "computational depth" of a 96-layer model with the parameter footprint of a 24-layer one.

However, there is a hidden cost: inference memory. During autoregressive decoding, Transformers store "keys" and "values" (the KV cache) to avoid recomputing the past. In a looped model, each iteration of the loop typically generates its own distinct KV states. If you loop 6 times, your KV cache becomes 6x larger. For long sequences, this memory bloat makes depth-recurrent models impractical for deployment. The authors ask: How can we reduce this memory overhead while keeping the benefits of exact attention?

Methods

The authors explore three primary strategies to tackle the memory-recurrence trade-off:

  1. Looped Hybrids: Instead of looping a stack of pure Transformer layers, they interleave "Transformer anchors" (standard attention) with recurrent sequence mixers like mLSTM or Gated DeltaNet. The mixers have a fixed-size state, meaning they don't add to the sequence-length-dependent KV cache.
  2. PARCAE-style Architectures: They test a design where recurrence is confined to a dedicated "latent workspace" between a fixed encoder (prelude) and decoder (coda). They specifically evaluate a "stabilizing adapter" that repeatedly re-injects the original input to prevent the recurrent state from drifting over many iterations.
  3. Recurrent KV Replay (Cache-Compression Training): This is the paper's most novel contribution. Instead of just trying to share KV caches at inference time (which usually degrades performance), they incorporate the sharing policy directly into training. They designate some iterations as Writers (which create new KV states) and others as Readers (which reuse the most recent Writer's state). This "KV Replay" teaches the model to remain functional even when it doesn't have unique memory for every layer of computation.

Datasets

The models were trained on the Nemotron-CC-HQ corpus (a high-quality web dataset) with a budget of 5.03B tokens (and one extended run at 29.6B tokens).

Evaluation was performed across a wide battery of standard benchmarks:

  • Common Sense & Knowledge: PIQA, WinoGrande, ARC (Easy/Challenge), HellaSwag, and SciQ.
  • Language Modeling: WikiText perplexity.
  • Reasoning: GSM8K (8-shot chain-of-thought).
  • CORE & CORE Ext: Benchmarks specifically designed to measure the benefits of increased computational depth.

Key Insights and Ideas

  • Training is Mandatory for Sharing: You cannot simply "squish" the KV cache at inference time if the model was trained with separate caches. "Inference-only compression" leads to massive performance drops. However, models trained with KV Replay recover almost all the performance of the full-memory versions.
  • Gated DeltaNet > mLSTM: In the looped hybrid setup, Gated DeltaNet proved significantly more robust than mLSTM. It handled larger gaps between attention anchors much better, suggesting it is a superior "bridge" for carrying information between attention layers.
  • The Power of Stride-3: The authors found that a "Stride-3" policy—where the model only writes new KV states every three iterations—offered a fantastic middle ground. It retains only 2 KV groups for 6 iterations of work, effectively tripling KV efficiency while matching the performance of a model with 6 unique KV groups.
  • Anchors are Essential: Looped hybrids need frequent attention anchors. If the "stride" between attention layers is too large, the recurrent mixers can't effectively reconstruct the exact token-level interactions needed for complex tasks.

Key Results

  • Performance Gains: A 6-iteration looped Transformer achieved a 18.96 score on CORE, significantly outperforming the non-recurrent 24-layer baseline (14.95).
  • Memory Efficiency: Using Recurrent KV Replay with Stride 3, the model maintained a score of 18.71 on CORE (nearly matching the full version) while reducing the analytical KV cache size from 576 KiB/token to 192 KiB/token.
  • Real-world Memory: Peak GPU memory delta during 8-shot GSM8K decoding dropped from 3241 MB (Stride 2) to 2922 MB (Stride 3), confirming that these architectural changes translate to actual hardware savings.
  • Hybrid Success: The Gated DeltaNet hybrid (main setup, $s_d=3$) achieved a CORE score of 18.79, proving that hybrids can rival pure looped Transformers while being even more memory-efficient.

Limitations and Future Work

  • Scale: The experiments were conducted on models with ~400M parameters. While the trends are promising, the authors note that the relative ordering of architectures might change at the multi-billion parameter scale.
  • Fixed Schedules: The KV sharing policies used were deterministic (e.g., "every 3rd iteration"). Future work could explore dynamic routing, where the model decides on-the-fly which tokens or iterations require a fresh KV write.
  • Reasoning Scores: The absolute scores on GSM8K were relatively low across all configurations due to the small model size and limited training budget. The authors treat these results as a comparative study of policies rather than a demonstration of state-of-the-art reasoning.

Summary updated

Abstract

Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework. We first study a looped hybrid that repeats a stack of Transformer layers, with recurrent mLSTM or Gated DeltaNet blocks replacing most attention layers and periodic Transformer attention anchors retained. We then restrict recurrence to a dedicated recurrent workspace in a Parcae-style architecture and test its stabilising adapter as a wrapper for the looped hybrid. Finally, we train a looped Transformer with recurrent KV replay, where only selected recurrence iterations write KV states that later iterations reuse. This trains the model to operate with the same shared KV topology used during inference. We find that looped hybrids require sufficiently frequent attention anchors, with performance depending strongly on the recurrent mixer. For cross-iteration KV sharing, recurrent KV replay allows multiple recurrence iterations to reuse the same KV states while retaining performance close to the full looped Transformer with separate KV caches for each iteration.
BibTeX
@misc{licakova2026architectures,
  author = {Lucia Licakova and Niklas Schmidinger and Lukas Hauzenberger and Sebastian Böck and Sepp Hochreiter},
  title = {{Architectures and KV Replay for Depth-Recurrent Language Models}},
  year = {2026},
  eprint = {2611.19446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19446},
}
Figure