TL;DR
State-Space Models (SSMs) like Mamba offer a highly efficient, linear-scaling alternative to Transformers, but they notoriously degrade when processing sequences longer than those seen during training. This paper demonstrates that this failure is driven by the eigenvalue spectrum of Mamba's state transition matrix $\mathbf{A}$, which causes the hidden state norm to explode or vanish over long contexts. By introducing a training-free "spectrum scaling" approach to modulate $\mathbf{A}$ layer-by-layer, the authors unlock robust long-context generalization without full model fine-tuning.
What Problem Does This Paper Address?
While Transformers dominate modern AI, their attention mechanism scales quadratically with sequence length, making long-context processing incredibly expensive. Mamba emerged as a powerful alternative, utilizing state-space dynamics to achieve linear time complexity. However, Mamba suffers from a major flaw: its performance collapses when forced to extrapolate to context lengths longer than its pre-training window.
Prior research attributed this vulnerability to the discretization time-steps ($\Delta_t$) failing to accumulate properly over long sequences. This paper challenges that assumption, shifting the blame to the out-of-distribution behavior of the state transition matrix $\mathbf{A}$. The authors set out to mathematically explain how these state dynamics break down at infinity and find a practical way to fix it.
Methods
The authors perform a rigorous spectral analysis of Mamba's continuous transition matrix $\mathbf{A}$. They prove theoretically that as input length approaches infinity, the expected squared norm of the hidden state becomes highly unstable based on the distribution of $\mathbf{A}$'s eigenvalues. Specifically, eigenvalues close to 1 cause state explosion, while eigenvalues close to 0 cause the state to vanish.
To counteract this, they propose Spectrum Scaling (Mamba Modulation). Instead of modifying the discretization step $\Delta_t$ (the standard practice in prior work), they directly scale the spectrum of the $\mathbf{A}$ matrices. They adapt a training-free calibration framework (MambaExtend) to learn optimal, constant scaling factors for $\mathbf{A}$ on a layer-by-layer basis. These factors are computed efficiently using either backpropagation or a black-box zeroth-order optimization technique (Simultaneous Perturbation Stochastic Approximation, or SPSA) on a tiny sample dataset (e.g., just 20 examples).
Datasets
The authors evaluated their approach across several standard long-context language modeling and retrieval benchmarks:
- Long-Context Language Modeling (Perplexity): Evaluated on the ProofPile, PG19, and GovReport datasets.
- Passkey Retrieval (Needle-in-a-Haystack): A synthetic task requiring the model to locate a 5-digit code randomly placed inside a large body of text from WikiText-103.
- LongBench: A comprehensive benchmark evaluating real-world long-context capabilities across tasks like Qasper, HotpotQA, 2WikiMultihopQA, TREC, TriviaQA, LCC, and RepoBench-P.
Key Insights and Ideas
- The Root Cause is Spectral: Visualizing the eigenvalues of pre-trained Mamba models reveals a delicate mix of local (rapidly decaying) and long-term (dominant) temporal modes. Long sequences push these dynamics out-of-distribution, causing the hidden state norm to diverge.
- Modulating $\mathbf{A}$ beats Modulating $\Delta_t$: While adjusting the discretization step $\Delta_t$ acts as a proxy to alter information decay, it doesn't fix the underlying eigenvalue instability. Directly scaling $\mathbf{A}$ addresses the root mathematical failure, providing a much more stable lever for length extension.
- Sensitivity to Model Scale: The failure of $\Delta_t$ scaling becomes drastically worse as models grow larger (e.g., Mamba2-2.7B), whereas scaling $\mathbf{A}$ consistently prevents perplexity explosion across all scales.
Key Results
- Drastic Perplexity Reductions: On long-context language modeling, calibrating $\mathbf{A}$ prevented the catastrophic perplexity explosions seen in the baseline models. For Mamba2-2.7B, scaling $\mathbf{A}$ maintained a stable perplexity that was up to 1000× lower than calibrating $\Delta_t$.
- Superior Downstream Performance: On the LongBench benchmark using Mamba2-2.7B, the Calibrated Scaling of $\mathbf{A}$ achieved an average score of 14.98, significantly outperforming both the uncalibrated Base Model (8.75) and the $\Delta_t$-scaling MambaExtend baseline (13.60).
Limitations and Future Work
The primary limitation acknowledged by the authors is that their work focuses strictly on Mamba and Mamba2 architectures. While the theoretical insights regarding state-space dynamics are broadly applicable, the exact modulation methodology requires adaptation for other linear recurrent or structured state-space models. Additionally, the evaluation was constrained by the limited availability of open-source, instruction-tuned Mamba variants. The authors suggest that exploring how spectrum scaling interacts with full instruction-tuning is a promising direction for future research.
Summary updated