CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models

Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke IwasawaarXiv 2026 · arXiv

TL;DR

This paper introduces CHASE (Cache-Hole-Adapted Skip Exit), a novel framework that successfully applies parameter-sharing loops and compute-saving early exits to State-Space Models (SSMs) like Mamba. By introducing a "cache-hole-consistent" training protocol, CHASE fixes the catastrophic perplexity degradation caused by skipping recurrent steps in SSMs, cutting inference compute roughly in half while maintaining high downstream accuracy.

What Problem Does This Paper Address?

Recent trends in efficient LLM design have popularized Looped Transformers, which reuse the same block across multiple layers to increase computational depth without blowing up the parameter count. To save compute at inference time, researchers use "early-exit" gates to stop processing easy tokens early.

However, whether this depth-recycling trick works for State-Space Models (SSMs) like Mamba remained an open question. More importantly, if you try to implement early exiting in an SSM by completely skipping deeper loops for a token, you run into a fatal architectural flaw: the cache-hole problem.

Unlike Transformers, which process tokens in parallel, SSMs rely on a continuous recurrent state passed from token to token. If token $A$ exits early and skips the remaining loops, it never writes its updates to the deeper recurrent states. When token $B$ arrives at those deeper loops, it encounters a "cache hole"—a broken state trajectory—causing the model's validation perplexity to explode.

Methods

The authors first formalize Looped Mamba and Looped Hybrid Mamba-Transformer architectures, denoted as $N \otimes R$ (where $N$ parameterized layers are looped $R$ times).

To enable actual compute savings via early exiting without breaking the recurrent state, they propose CHASE:

  1. Adaptive Exit Gate: A lightweight token-level gate (adapted from the Ouro framework) that calculates the probability of exiting at any given recurrent step based on the token's hidden state.
  2. Cache-Hole-Consistent Forward Pass: During training, CHASE stochastically simulates early exits. Tokens that "exit" are frozen and packed out of subsequent deeper loops.
  3. Exit-Depth Supervision: The model is trained on these "holed" states, forcing the backbone to learn how to process subsequent tokens even when previous tokens skipped updates.
Illustration of the cache-hole problem and how CHASE bypasses skipped states to keep trajectories consistent.

Datasets

The authors evaluated their approach across synthetic reasoning tasks and standard language modeling benchmarks:

  • Mano Task: A synthetic modular-arithmetic manipulation task testing tree-structured parsing and knowledge composition.
  • p-hop Induction: A multi-step generalization of the induction-head problem testing recursive contextual retrieval.
  • Language Model Pre-training: Trained on the Nemotron-ClimbMix dataset (100B tokens for base comparisons, 150B tokens for exit-gate variants).
  • Downstream Benchmarks: PIQA, BoolQ, WinoGrande, OpenBookQA, HellaSwag, ARC-Challenge, and ARC-Easy.

Key Insights and Ideas

  • Recurrent Depth Matters for SSMs: Just like Transformers, Mamba models benefit significantly from extra computational depth via looping, outperforming parameter-matched non-looped baselines on complex reasoning tasks.
  • Naively Skipping Loops is Catastrophic: In looped Transformers, early exiting is often simulated by just changing the readout layer while still running the full compute. In SSMs, trying to save actual compute by skipping loops breaks the state chain, causing perplexity to skyrocket (up to +4788%).
  • Alignment is the Cure: By explicitly training the Mamba backbone on state trajectories that contain cache holes, CHASE aligns the training distribution with the actual inference-skipping behavior, stabilizing the model.

Key Results

  • Reasoning Superiority: On the Mano task ($L=24$), a shallow 4-layer Mamba model looped 6 times ($4 \otimes 6$) achieved 99.35% accuracy, completely crushing its parameter-matched non-looped baseline ($4 \otimes 1$), which scored just 7.50%. It even beat a deep 24-layer non-looped model (37.45%).
Accuracy comparison on the Mano task showing looped models outperforming non-looped baselines.
  • Perplexity Stability: While standard early-exiting (Ouro Stage-2 + Skip) caused validation perplexity to explode as compute was reduced, CHASE maintained a stable perplexity within 3–4% of full computation.
Validation perplexity vs compute showing CHASE's stability compared to the stock Ouro gate under skip inference.
  • Compute Savings & Speedups: CHASE matched or exceeded full-compute performance on downstream tasks while executing roughly half the recurrent steps. This translated into a 1.73x to 1.83x wall-clock prefill speedup at a batch size of 64 on an NVIDIA RTX 5090.

Limitations and Future Work

  • Scale Limitations: The experiments were capped at a maximum model size of 370M parameters and a pre-training budget of 150B tokens. Scaling behavior at the billion-parameter mark remains unverified.
  • Hybrid Model KV-Caches: While CHASE works perfectly for pure Looped Mamba backbones, it does not yet handle the key-value (KV) cache holes introduced by the attention layers in Hybrid Mamba-Transformer models. The authors leave KV-aware cache-hole adaptation to future work.

Summary updated

Abstract

Recent work on looped language models suggests that many reasoning problems benefit from greater computational depth rather than from additional independent parameters. Existing studies, however, focus almost exclusively on Transformer backbones, leaving open whether this principle also applies to state-space language models. We investigate Looped Mamba and Looped Hybrid Mamba-Transformer architectures, which repeatedly apply a shared Mamba (or hybrid) block to introduce explicit finite-depth recurrent computation. On two controlled reasoning tasks-Mano (modular-arithmetic manipulation) and p-hop induction-Looped Mamba consistently outperforms parameter-matched non-looped baselines and, in several settings, matches or exceeds non-looped models of equal effective depth. We then extend the study to language model pre-training under matched iso-parameter and iso-FLOPs protocols, which jointly disentangle the effects of parameter sharing and effective depth: looped models remain competitive on downstream benchmarks with substantially fewer distinct parameters, although deeper non-looped models retain an advantage in validation perplexity under strict iso-FLOPs comparisons. Finally, we adapt Ouro's two-stage exit gate to Looped Mamba for threshold-controlled selection among recurrent-step outputs. Executing such exits on a state-space backbone, however, leaves the recurrent state without its deeper updates, and validation perplexity then degrades severely. We therefore introduce a cache-hole adaptation that aligns continued training with skipped-state inference. At the scales studied, the adapted model keeps perplexity close to full computation and matches or exceeds full-compute exit-state selection on downstream benchmarks while executing roughly half of the recurrent steps, which translates into measured inference speedups once the prefill is compute-bound.
BibTeX
@misc{yu2026chase,
  author = {Zhenxuan Yu and Takeshi Kojima and Yutaka Matsuo and Yusuke Iwasawa},
  title = {{CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models}},
  year = {2026},
  eprint = {2607.10110},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.10110},
}
Figure