Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing

Peihao Wang, Ruisi Cai, Yuehao Wang, Jiajun Zhu, Pragya Srivastava, Zhangyang Wang, Pan Li

TL;DR

While State Space Models (SSMs) like Mamba are celebrated for their efficiency with long sequences, they suffer from a fundamental dilemma: an inherent recency bias that causes them to forget distant information and an over-smoothing effect that makes token representations indistinguishable in deeper architectures. This paper introduces Polarization, a simple yet effective technique that fixes two state transition channels to 0 and 1, successfully mitigating both bottlenecks and unlocking the performance of deeper SSMs.

What Problem Does This Paper Address?

Modern sequence modeling has seen a shift from Transformers to Structured State Space Models (SSMs) due to the latter's linear scaling and constant-memory inference. However, despite their promise of "infinite" context, SSMs often underperform Transformers on long-range tasks.

The authors identify two core limitations:

  1. Recency Bias: Theoretically and empirically, the influence of a token on future outputs decays exponentially with distance. This makes SSMs "locally-minded," struggling to recall information from the beginning of a long document.
  2. Over-smoothing: To capture longer context, one might simply add more layers. However, the authors prove that SSM layers act as low-pass filters. As depth increases, token representations collapse toward a mean, becoming indistinguishable and rendering the model's state uninformative.

Methods

The researchers first provide a rigorous theoretical framework (Theorem 3.1 and 4.2) to prove that recency bias and over-smoothing are intrinsic to the SSM architecture.

To solve this, they propose Polarization. In an SSM, the state transition is governed by a matrix $A_t$. The authors suggest "polarizing" two specific channels in this matrix:

  • The "One" Channel: One channel is fixed to exactly 1. This acts as a perfect memory, preserving historical information without decay, thus fighting recency bias.
  • The "Zero" Channel: Another channel is fixed to exactly 0. This ensures the model always has a "fresh" look at the current token, preventing it from being completely smoothed out by past information.

This technique is remarkably simple to implement—requiring only a small modification to the forward pass—yet it fundamentally changes the information flow dynamics.

Datasets

The authors evaluated their findings and the Polarization technique using several benchmarks:

  • Needle in a Haystack: A retrieval task where a specific fact is buried at various positions in a long document (up to 256k tokens).
  • Associative Recall (AR): A synthetic task requiring the model to remember key-value pairs and retrieve the value associated with a specific key.
  • CIFAR-10 (Pixel-level): Images flattened into sequences of 1,024 pixels to test robustness and locality bias.
  • DataComp-LM: Used for large-scale pre-training experiments to observe scaling laws with depth and context length.

Key Insights and Ideas

  • The Recency Trap: Influence scores in SSMs decay linearly in log-space (exponentially in real space). As shown in Figure 1, even pre-trained models like Mamba-1.4B exhibit this steep decay, confirming that the bias is architectural, not just a result of poor training. Influence scores decay exponentially with distance
  • Adversarial Vulnerability: Because SSMs over-weight recent tokens, they are uniquely vulnerable to "jailbreak" or backdoor attacks. An attacker can append malicious instructions to the end of a prompt to override earlier system instructions, a tactic that is much less effective against Transformers.
  • The Depth Bottleneck: While deeper models should have a larger receptive field, Figure 4 shows that performance eventually plateaus and then declines as depth increases due to feature over-smoothing.
  • Polarization as a Unified Fix: By manually setting the "extremes" of the transition matrix (0 and 1), the model is forced to maintain both high-frequency (current token) and zero-frequency (long-term history) information, bypassing the natural low-pass filtering of standard SSM layers.
Figure 1: Visualization of log influential scores log |∂yt/∂xs| versus distance (t−s).

Key Results

The "Needle in a Haystack" test highlights the stark difference between architectures. While Transformers maintain stable accuracy regardless of where the "needle" is placed, SSMs show a massive performance drop when the information is at the start of the document (the "left" side of the heatmap).

Figure 2: Comparison between SSM and Transformer on the “Needle in a Haystack" benchmark. The left figure shows the retrieval accuracy of the Mamba-Codestral-7B model, while the right figure presents the retrieval accuracy of the Mistral-7B model. We present a heatmap where "full context length" refers to the total length of the document, and "needle position" denotes the relative position of the statement to be retrieved within the context. See more fine-grained visualization in Appendix E.2.

Quantitatively, the Polarization technique showed impressive gains in Associative Recall:

  • A standard 4-layer Mamba model achieved 71.61% average accuracy.
  • By adding one- and zero-polarized channels, the same 4-layer model jumped to 93.43% accuracy.
  • Crucially, Polarization allowed the model to actually benefit from depth; without it, increasing layers from 2 to 4 often resulted in a performance drop on long-range tasks.

Limitations and Future Work

The authors acknowledge that while Polarization addresses the architectural bottlenecks, it does not fundamentally change the fact that SSMs must compress history into a fixed-size hidden state. Future work could explore more complex "structured" polarization or investigate how these findings apply to hybrid models that combine SSMs with attention mechanisms. Additionally, the study primarily focuses on real-valued parameterizations; while they touch on complex-valued SSMs, further empirical validation in that space is warranted.

Summary updated

Abstract

Structured State Space Models (SSMs) have emerged as alternatives to transformers. While SSMs are often regarded as effective in capturing long-sequence dependencies, we rigorously demonstrate that they are inherently limited by strong recency bias. Our empirical studies also reveal that this bias impairs the models' ability to recall distant information and introduces robustness issues. Our scaling experiments then discovered that deeper structures in SSMs can facilitate the learning of long contexts. However, subsequent theoretical analysis reveals that as SSMs increase in depth, they exhibit another inevitable tendency toward over-smoothing, e.g., token representations becoming increasingly indistinguishable. This fundamental dilemma between recency and over-smoothing hinders the scalability of existing SSMs. Inspired by our theoretical findings, we propose to polarize two channels of the state transition matrices in SSMs, setting them to zero and one, respectively, simultaneously addressing recency bias and over-smoothing. Experiments demonstrate that our polarization technique consistently enhances the associative recall accuracy of long-range tokens and unlocks SSMs to benefit further from deeper architectures. All source codes are released at https://github.com/VITA-Group/SSM-Bottleneck.
BibTeX
@inproceedings{wang2025understanding,
  author = {Peihao Wang and Ruisi Cai and Yuehao Wang and Jiajun Zhu and Pragya Srivastava and Zhangyang Wang and Pan Li},
  title = {{Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=pymXpl4qvi},
}
Figure