Abstract: Tamanna Hossain, Robert L. Logan Iv, Chandrasekhara Ganesh Jagadeesan, Sameer Singh, Joel R. Tetreault, Alejandro Jaimes. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.
2025.
Introduction
The landscape of Large Language Model (LLM) architectures is currently defined by a tension between the expressive power of Transformers and the efficiency of State Space Models (SSMs). While Transformers excel at capturing long-range dependencies, their memory requirements scale linearly with sequence length due to the Key-Value (KV) cache. SSMs, such as Mamba, address this by utilizing a fixed-size hidden state, enabling constant-time inference and significantly reduced memory overhead for long contexts.
However, this efficiency comes at a cost: a finite hidden state cannot store an infinite amount of information. While theoretical work has established that SSMs must eventually discard information, the specific nature of this "selective amnesia" has remained largely uncharacterized. This paper investigates what types of information Mamba prioritizes and what it chooses to forget during long-context processing.
Methodology: Probing via Auto-Encoders
To analyze the contents of Mamba's hidden states, the researchers developed a probing technique using auto-encoders. The approach involves three main components:
- Frozen Encoder: A pre-trained Mamba model (ranging from 130M to 1.4B parameters) serves as the encoder. Its weights are frozen to ensure the probe measures existing internal representations rather than inducing new learning.
- Trainable Decoder: A decoder, sharing the Mamba architecture, is trained to reconstruct the original input text solely from the final hidden state of the frozen encoder.
- Information Reconstruction: By comparing the original input to the reconstructed output, the researchers can quantify information loss. If a specific token cannot be reconstructed, it is inferred that the token was not preserved in the hidden state.
Performance is evaluated using ROUGE F1 for overall sequence similarity and Omission Rate to track the frequency of specific token types being lost.
Key Findings: The Selective Memory of SSMs
The study reveals several systematic biases in how Mamba manages its limited memory:
The "Numerical Black Hole"
One of the most striking findings is Mamba's difficulty with numerical data. Across all model sizes, numbers (NUM) exhibit the highest omission rates compared to any other part of speech. In the 130M parameter model, the omission rate for numbers exceeded 50%. While scaling to 1.4B parameters improves overall reconstruction, the relative difficulty of retaining numbers persists.
Recency Bias and Structural Limits
The models exhibit a pronounced recency bias. In 256-token sequences, tokens at the beginning were forgotten up to six times more frequently than those at the end. Furthermore, synthetic tests showed that while Mamba can often identify which token was repeated in a sequence, it struggles to remember the exact count of repetitions for sequences longer than four tokens.
Dialect Disparity
The research identifies a significant performance gap when processing different dialects. Information written in African American Vernacular English (AAVE) is forgotten significantly more often than Standard American English (SAE). This suggests that Mamba's compression heuristics may be biased toward the dominant linguistic structures found in its pre-training data.
Discussion and Implications
The "Rarity Hypothesis" proposed by the authors suggests that Mamba prioritizes tokens that are common and predictable (like determiners) over those with a "long tail" of unique values, such as numbers or specific named entities.
While scaling model parameters improves reconstruction quality (ROUGE F1 increases from ~67 to ~80 for 256-token sequences when moving from 130M to 1.4B), the fundamental patterns of forgetting remain consistent. This indicates that the memory bottleneck is an architectural characteristic rather than a simple lack of capacity.
For practitioners, these results provide a crucial warning: when deploying SSMs for tasks requiring high precision in mathematics, code, or diverse dialects, the model's fixed-size memory may systematically discard the most critical information. Future work may need to explore alternative tokenization strategies or hybrid architectures to mitigate these selective memory failures.
Summary updated