TL;DR
This paper introduces CHASE (Cache-Hole-Adapted Skip Exit), a novel framework that successfully applies parameter-sharing loops and compute-saving early exits to State-Space Models (SSMs) like Mamba. By introducing a "cache-hole-consistent" training protocol, CHASE fixes the catastrophic perplexity degradation caused by skipping recurrent steps in SSMs, cutting inference compute roughly in half while maintaining high downstream accuracy.
What Problem Does This Paper Address?
Recent trends in efficient LLM design have popularized Looped Transformers, which reuse the same block across multiple layers to increase computational depth without blowing up the parameter count. To save compute at inference time, researchers use "early-exit" gates to stop processing easy tokens early.
However, whether this depth-recycling trick works for State-Space Models (SSMs) like Mamba remained an open question. More importantly, if you try to implement early exiting in an SSM by completely skipping deeper loops for a token, you run into a fatal architectural flaw: the cache-hole problem.
Unlike Transformers, which process tokens in parallel, SSMs rely on a continuous recurrent state passed from token to token. If token $A$ exits early and skips the remaining loops, it never writes its updates to the deeper recurrent states. When token $B$ arrives at those deeper loops, it encounters a "cache hole"—a broken state trajectory—causing the model's validation perplexity to explode.
Methods
The authors first formalize Looped Mamba and Looped Hybrid Mamba-Transformer architectures, denoted as $N \otimes R$ (where $N$ parameterized layers are looped $R$ times).
To enable actual compute savings via early exiting without breaking the recurrent state, they propose CHASE:
- Adaptive Exit Gate: A lightweight token-level gate (adapted from the Ouro framework) that calculates the probability of exiting at any given recurrent step based on the token's hidden state.
- Cache-Hole-Consistent Forward Pass: During training, CHASE stochastically simulates early exits. Tokens that "exit" are frozen and packed out of subsequent deeper loops.
- Exit-Depth Supervision: The model is trained on these "holed" states, forcing the backbone to learn how to process subsequent tokens even when previous tokens skipped updates.
Datasets
The authors evaluated their approach across synthetic reasoning tasks and standard language modeling benchmarks:
- Mano Task: A synthetic modular-arithmetic manipulation task testing tree-structured parsing and knowledge composition.
- p-hop Induction: A multi-step generalization of the induction-head problem testing recursive contextual retrieval.
- Language Model Pre-training: Trained on the Nemotron-ClimbMix dataset (100B tokens for base comparisons, 150B tokens for exit-gate variants).
- Downstream Benchmarks: PIQA, BoolQ, WinoGrande, OpenBookQA, HellaSwag, ARC-Challenge, and ARC-Easy.
Key Insights and Ideas
- Recurrent Depth Matters for SSMs: Just like Transformers, Mamba models benefit significantly from extra computational depth via looping, outperforming parameter-matched non-looped baselines on complex reasoning tasks.
- Naively Skipping Loops is Catastrophic: In looped Transformers, early exiting is often simulated by just changing the readout layer while still running the full compute. In SSMs, trying to save actual compute by skipping loops breaks the state chain, causing perplexity to skyrocket (up to +4788%).
- Alignment is the Cure: By explicitly training the Mamba backbone on state trajectories that contain cache holes, CHASE aligns the training distribution with the actual inference-skipping behavior, stabilizing the model.
Key Results
- Reasoning Superiority: On the Mano task ($L=24$), a shallow 4-layer Mamba model looped 6 times ($4 \otimes 6$) achieved 99.35% accuracy, completely crushing its parameter-matched non-looped baseline ($4 \otimes 1$), which scored just 7.50%. It even beat a deep 24-layer non-looped model (37.45%).
- Perplexity Stability: While standard early-exiting (Ouro Stage-2 + Skip) caused validation perplexity to explode as compute was reduced, CHASE maintained a stable perplexity within 3–4% of full computation.
- Compute Savings & Speedups: CHASE matched or exceeded full-compute performance on downstream tasks while executing roughly half the recurrent steps. This translated into a 1.73x to 1.83x wall-clock prefill speedup at a batch size of 64 on an NVIDIA RTX 5090.
Limitations and Future Work
- Scale Limitations: The experiments were capped at a maximum model size of 370M parameters and a pre-training budget of 150B tokens. Scaling behavior at the billion-parameter mark remains unverified.
- Hybrid Model KV-Caches: While CHASE works perfectly for pure Looped Mamba backbones, it does not yet handle the key-value (KV) cache holes introduced by the attention layers in Hybrid Mamba-Transformer models. The authors leave KV-aware cache-hole adaptation to future work.
Summary updated