TL;DR
While State Space Models (SSMs) like Mamba are celebrated for their efficiency with long sequences, they suffer from a fundamental dilemma: an inherent recency bias that causes them to forget distant information and an over-smoothing effect that makes token representations indistinguishable in deeper architectures. This paper introduces Polarization, a simple yet effective technique that fixes two state transition channels to 0 and 1, successfully mitigating both bottlenecks and unlocking the performance of deeper SSMs.
What Problem Does This Paper Address?
Modern sequence modeling has seen a shift from Transformers to Structured State Space Models (SSMs) due to the latter's linear scaling and constant-memory inference. However, despite their promise of "infinite" context, SSMs often underperform Transformers on long-range tasks.
The authors identify two core limitations:
- Recency Bias: Theoretically and empirically, the influence of a token on future outputs decays exponentially with distance. This makes SSMs "locally-minded," struggling to recall information from the beginning of a long document.
- Over-smoothing: To capture longer context, one might simply add more layers. However, the authors prove that SSM layers act as low-pass filters. As depth increases, token representations collapse toward a mean, becoming indistinguishable and rendering the model's state uninformative.
Methods
The researchers first provide a rigorous theoretical framework (Theorem 3.1 and 4.2) to prove that recency bias and over-smoothing are intrinsic to the SSM architecture.
To solve this, they propose Polarization. In an SSM, the state transition is governed by a matrix $A_t$. The authors suggest "polarizing" two specific channels in this matrix:
- The "One" Channel: One channel is fixed to exactly 1. This acts as a perfect memory, preserving historical information without decay, thus fighting recency bias.
- The "Zero" Channel: Another channel is fixed to exactly 0. This ensures the model always has a "fresh" look at the current token, preventing it from being completely smoothed out by past information.
This technique is remarkably simple to implement—requiring only a small modification to the forward pass—yet it fundamentally changes the information flow dynamics.
Datasets
The authors evaluated their findings and the Polarization technique using several benchmarks:
- Needle in a Haystack: A retrieval task where a specific fact is buried at various positions in a long document (up to 256k tokens).
- Associative Recall (AR): A synthetic task requiring the model to remember key-value pairs and retrieve the value associated with a specific key.
- CIFAR-10 (Pixel-level): Images flattened into sequences of 1,024 pixels to test robustness and locality bias.
- DataComp-LM: Used for large-scale pre-training experiments to observe scaling laws with depth and context length.
Key Insights and Ideas
- The Recency Trap: Influence scores in SSMs decay linearly in log-space (exponentially in real space). As shown in Figure 1, even pre-trained models like Mamba-1.4B exhibit this steep decay, confirming that the bias is architectural, not just a result of poor training. Influence scores decay exponentially with distance
- Adversarial Vulnerability: Because SSMs over-weight recent tokens, they are uniquely vulnerable to "jailbreak" or backdoor attacks. An attacker can append malicious instructions to the end of a prompt to override earlier system instructions, a tactic that is much less effective against Transformers.
- The Depth Bottleneck: While deeper models should have a larger receptive field, Figure 4 shows that performance eventually plateaus and then declines as depth increases due to feature over-smoothing.
- Polarization as a Unified Fix: By manually setting the "extremes" of the transition matrix (0 and 1), the model is forced to maintain both high-frequency (current token) and zero-frequency (long-term history) information, bypassing the natural low-pass filtering of standard SSM layers.
Key Results
The "Needle in a Haystack" test highlights the stark difference between architectures. While Transformers maintain stable accuracy regardless of where the "needle" is placed, SSMs show a massive performance drop when the information is at the start of the document (the "left" side of the heatmap).
Quantitatively, the Polarization technique showed impressive gains in Associative Recall:
- A standard 4-layer Mamba model achieved 71.61% average accuracy.
- By adding one- and zero-polarized channels, the same 4-layer model jumped to 93.43% accuracy.
- Crucially, Polarization allowed the model to actually benefit from depth; without it, increasing layers from 2 to 4 often resulted in a performance drop on long-range tasks.
Limitations and Future Work
The authors acknowledge that while Polarization addresses the architectural bottlenecks, it does not fundamentally change the fact that SSMs must compress history into a fixed-size hidden state. Future work could explore more complex "structured" polarization or investigate how these findings apply to hybrid models that combine SSMs with attention mechanisms. Additionally, the study primarily focuses on real-valued parameterizations; while they touch on complex-valued SSMs, further empirical validation in that space is warranted.
Summary updated