Demystifying the Token Dynamics of Deep Selective State Space Models

Thieu Vo, Duy-Tung Pham, Xin T. Tong, Tan Minh Nguyen

TL;DR

This paper dives into the theoretical underpinnings of Mamba, a popular selective state space model (SSM), by analyzing the continuous-time dynamics of its "tokens" (internal representations). They discover that tokens either all vanish or all explode, and propose practical ways to leverage this understanding to improve Mamba's performance.

What Problem Does This Paper Address?

Selective State Space Models (SSMs), like Mamba, have shown remarkable success in processing sequential data, often outperforming traditional Transformers in efficiency and long-range dependency modeling. However, despite their empirical prowess, a deep theoretical understanding of why they work so well, particularly the internal dynamics of how information flows and transforms through the model, has been lacking. This theoretical gap hinders further development and reliable application in high-fidelity tasks. This paper aims to bridge this gap by analyzing the dynamical behavior of tokens within a Mamba model.

Methods

The authors tackle this by deriving a continuous-time dynamical system that represents the core "S6 layer" of Mamba. This allows them to analyze the long-term behavior of the model's internal representations (tokens) as a function of its parameters. They focus on the one-dimensional case (single feature channel) to derive theoretical guarantees. Key techniques include:

  • Continuous-Time Limit: Transforming the discrete layer-by-layer processing of Mamba into a continuous-time differential equation.
  • Asymptotic Analysis: Studying the behavior of the solutions to this differential equation as time (representing model depth) goes to infinity.
  • Parameter Analysis: Identifying specific parameter regimes (related to the input-output projection matrix $S_C S_B$) that dictate whether tokens converge or diverge.
  • Empirical Validation: Conducting experiments to confirm theoretical findings and test proposed model refinements.

Datasets

  • WIKITEXT103: Used for auto-regressive language modeling experiments.
  • ImageNet-1K: Used for image classification experiments.

Key Insights and Ideas

  • Binary Token Fate: In the simplified one-dimensional Mamba, tokens are not nuanced; they either all converge to zero (effectively vanishing) or all diverge to infinity. There's no middle ground where some tokens stabilize while others grow.
  • Parameter Dictates Fate: The behavior of tokens is directly controlled by the sign of the eigenvalues of a specific matrix derived from the model's parameters ($S_C S_B$). Negative eigenvalues lead to convergence, while positive eigenvalues lead to divergence.
  • Convergence is Bad: The "convergence scenario" (where all tokens go to zero) is detrimental to model performance. This suggests that for effective learning, tokens need to retain and amplify information, not disappear.
  • Unequal Divergence: In the "divergence scenario," tokens don't just explode uniformly. They diverge at different rates, implying that some tokens are more "important" than others during training. This unequal contribution is a key insight for optimization.
  • Higher Dimensions are Complex: While the one-dimensional case is clear-cut, the authors conjecture that in higher dimensions, the dynamics are determined by the symmetric part of the $S_C S_B$ matrix, with negative eigenvalues still leading to convergence and at least one positive eigenvalue leading to divergence.

Key Results

  • Theoretical Guarantees: Proved that in the 1D case, tokens either converge to zero (at a rate of $O(1/\sqrt{t})$) or diverge to infinity.
  • Parameter Criteria: Provided explicit conditions based on model parameters ($S_C S_B$) to predict convergence or divergence.
  • Performance Impact: Empirically showed that the convergence scenario negatively impacts language modeling performance (higher perplexity).
  • Refinement 1 (Exclusion): Removing the convergence scenario by constraining parameters led to improved performance.
  • Refinement 2 (Reordering): Reordering tokens based on their divergence rates (importance scores) improved accuracy and convergence speed on ImageNet-1K and language modeling tasks. For example, on ImageNet-1K, token reordering boosted Top-1 accuracy from 81.90% to 82.02% and Top-5 from 95.86% to 95.87%. On WIKITEXT103, reordering reduced perplexity by 1.49 for the Mamba model.

Limitations and Future Work

  • 1D Focus: The theoretical analysis is primarily for the one-dimensional case. Extending this to higher dimensions is a significant open problem, though the authors offer a conjecture.
  • Computational Overhead: The proposed token reordering method introduces a small computational overhead during training.
  • Optimality of Reordering: The authors are uncertain if their specific reordering strategy (based on orthogonal projection) is globally optimal.
  • Other Components: The analysis focuses on the S6 layer and doesn't deeply consider the impact of other components like layer normalization or convolutions on token dynamics, though they acknowledge these will be explored in future work.
  • Higher-Dimensional Dynamics: A systematic study of token dynamics in higher-dimensional cases is left for future research.

Summary updated

Abstract

Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity. We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance. For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training. Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance. Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications.
BibTeX
@inproceedings{vo2025demystifying,
  author = {Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},
  title = {{Demystifying the Token Dynamics of Deep Selective State Space Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=qtTIP5Gjc5},
}
Figure