Abstract
State space models (SSMs), such as Mamba, have emerged as an efficient alternative to transformers for long-context sequence modeling. However, despite their growing adoption, SSMs lack the interpretability tools that have been crucial for understanding and improving attention-based architectures. While recent efforts provide insights into Mamba's internal mechanisms, they do not explicitly decompose token-wise contributions, leaving gaps in understanding how Mamba selectively processes sequences across layers. In this work, we introduce LaTIM, a novel token-level decomposition method for both Mamba-1 and Mamba-2 that enables fine-grained interpretability. We extensively evaluate our method across diverse tasks, including machine translation, copying, and retrieval-based generation, demonstrating its effectiveness in revealing Mamba's token-to-token interaction patterns.
Introduction
Transformers have dominated the landscape of large language models (LLMs) primarily due to the transparency of the attention mechanism. Attention maps provide a literal representation of how specific input tokens influence output predictions, serving as a vital tool for debugging and architectural refinement. In contrast, State Space Models (SSMs) like Mamba offer superior efficiency for long sequences by utilizing a compressed hidden state that evolves over time. However, this recurrence-based approach makes Mamba a "black box," as it lacks the explicit token-to-token maps found in Transformers.
To address this gap, we present LATIM (Latent Token-to-Token Interactions), a framework designed to mathematically decompose Mamba’s internal state updates. By unrolling the recurrence and approximating non-linearities, LATIM allows researchers to visualize how information flows from token to token, revealing the underlying logic of SSM-based architectures.
Methodology
The core challenge in interpreting Mamba lies in its non-linear components—specifically the SiLU activation and 1D convolutions—which mix information in a non-additive manner. LATIM overcomes this through several key technical innovations:
- Unrolling the Recurrence: We reformulate the SSM's linear recurrence into a "hidden attention" matrix. This matrix represents the contribution of each previous state to the current output, effectively creating a pseudo-attention map for the recurrent structure.
- Approximating Non-Linearity: Since the SiLU activation is not naturally additive, we approximate it with an additive function $f$. Empirical testing revealed that using SiLU itself as the approximation function yielded the lowest error, maintaining high faithfulness to the model's actual behavior.
- Mamba-2 Decomposition: For the Mamba-2 architecture, we treat the GroupNorm layer as a local affine transformation. This allows us to maintain token-wise traceability even through the more complex normalization layers of the newer architecture.
- Attribution Variants: We adapted established Transformer interpretability techniques, such as ALTI (measuring vector distances) and ALTI-Logit (tracing contributions to final predictions), to function within the LATIM framework.
Experimental Setup
We evaluated LATIM across three distinct task categories to ensure its robustness and utility:
- Synthetic Copying: A benchmark requiring the model to repeat a specific string. This task provides a "ground truth" diagonal attention pattern, allowing us to verify if LATIM accurately captures known information flows.
- Machine Translation (MT): Using the IWSLT17 datasets (German↔English and French→English), we measured the Alignment Error Rate (AER) to determine if LATIM's interaction maps align with human-provided word alignments.
- Retrieval-Based Generation: Utilizing the RULER benchmark, we focused on Passkey Retrieval (finding a hidden number in a large context) and Frequent Word Extraction (counting synthetic words).
Key Insights and Results
The "Decaying Attention" Phenomenon
LATIM revealed a critical insight into why SSMs sometimes struggle with precise counting tasks. In Frequent Word Extraction, we observed that Mamba’s focus on repeated words decays over time. The model "attends" heavily to the first few instances of a word but loses interest in subsequent occurrences. This visual evidence explains the performance drop-off in long-range counting tasks.
First-Key Bias
In multi-key retrieval tasks, LATIM visualizations showed a significant bias toward the first key appearing in the context. This provides a clear mechanical explanation for why Mamba's accuracy diminishes as the number of keys in the context increases.
Faithfulness and Alignment
LATIM demonstrates high quantitative faithfulness to the model's internal logic:
- Copying Task: LATIM (ℓ₂) achieved an AUC of 0.98 for Mamba-2, significantly outperforming the Mamba-Attention baseline (0.79).
- Translation Alignment: In German→English translation, LATIM achieved an AER of 0.44, a substantial improvement over Mamba-Attention (0.84) and MambaLRP (0.47).
- Sparsity: Unlike previous methods that produced blurry "clouds" of importance, LATIM generates sparse, linguistically relevant heatmaps that are more intuitive for human analysis.
Discussion and Limitations
While LATIM provides a powerful lens into Mamba, it is not without limitations. The SiLU approximation, while effective, introduces a small degree of error (as low as 0.21 per layer) that might obscure extremely subtle behaviors in high-stakes mechanistic interpretability scenarios. Furthermore, our current evaluation focused on translation and retrieval; more complex reasoning tasks may require further validation.
Future work will explore the integration of LATIM into hybrid models (like Jamba) that combine Mamba and Transformer layers. Additionally, we aim to use the insights gained from LATIM to develop more robust SSM architectures that are less susceptible to biases like the first-key effect.
Conclusion
LATIM bridges the interpretability gap between Transformers and SSMs. By providing a reliable method for token-level decomposition, it transforms Mamba from a black-box recurrent model into a transparent architecture where information flow can be measured, visualized, and eventually optimized.
Summary updated