Demystifying the Token Dynamics of Deep Selective State Space Models

Thieu N Vo, Tung D. Pham, Xin T. Tong, Tan Minh Nguyen

Selective State Space Models (SSMs), particularly the Mamba architecture, have taken the AI world by storm by offering Transformer-like performance with linear scaling. However, while they work exceptionally well in practice, we haven't fully understood why from a theoretical standpoint.

In a new paper accepted at ICLR 2025, researchers from the National University of Singapore and FPT Software AI Center pull back the curtain on Mamba’s inner workings. By treating the model as a continuous-time dynamical system, they reveal how tokens evolve as they pass through deep layers, leading to practical refinements that boost performance in both language and vision tasks.

TL;DR

This paper derives the continuous-time limit of the Mamba model to prove that tokens either vanish to zero or explode to infinity as they go deeper. By ensuring tokens diverge and reordering them based on a novel "importance score," the authors significantly improve Mamba's accuracy and training speed.

What Problem Does This Paper Address?

Despite their success, deep selective SSMs remain somewhat of a "black box." Specifically, we lack a comprehensive theoretical understanding of how tokens—the individual units of data like words or image patches—change as they move through the model's layers.

In Transformers, token dynamics are known to exhibit exponential divergence. In Mamba, the "selective" nature (where parameters change based on the input) makes the math much harder. The authors sought to answer: What is the long-term fate of a token in a deep Mamba model, and how does that fate affect the model's ability to learn?

Methods

The researchers approached Mamba through the lens of Neural ODEs. They derived a continuous-time dynamical system that represents the Mamba model as the number of layers goes to infinity.

Key technical steps included:

  1. Continuous-Time Derivation: They simplified the S6 layer (Mamba's core) into a system of differential equations.
  2. Asymptotic Analysis: They analyzed the stability of these equations to see if solutions (tokens) converge or diverge over "time" (model depth).
  3. Eigenvalue Analysis: They identified that the symmetric part of the input-output matrix ($S_C^\top S_B$) acts as a "switch" for the model's behavior.
  4. Token Reordering: Based on their findings that tokens diverge at different rates, they proposed a SoftSort-based reordering mechanism. This ranks tokens by an "importance score" before they enter the S6 layer, ensuring the most critical data is prioritized during training.

Datasets

The authors validated their theoretical findings across several standard benchmarks:

  • Language Modeling: WikiText-103 (using a 130M parameter Mamba model).
  • Image Classification: ImageNet-1K (using the MambaVision-T baseline).
  • Text Classification: IMDB and AGNews.

Key Insights and Ideas

  • The Binary Fate of Tokens: In the one-dimensional case, tokens follow an "all or nothing" rule. Depending on the model parameters, either all tokens converge to zero (vanishing) or all tokens diverge to infinity (exploding).
  • Convergence is a Performance Killer: When tokens converge to zero, the model loses diversity and randomness, which the authors empirically proved leads to higher perplexity (worse performance) in language tasks.
  • Unequal Contribution: In the divergence scenario, tokens don't explode at the same speed. Tokens with higher initial values diverge faster. This means the model naturally "prioritizes" certain tokens over others during the update process.
  • The Importance Switch: The sign of the eigenvalues of the input-output matrix determines the scenario. Positive eigenvalues lead to the beneficial divergence scenario, while negative ones lead to the harmful convergence scenario.
Table 1: Summary of the dynamical properties of this work.

Key Results

The theoretical refinements led to measurable gains:

  • Language Modeling: On WikiText-103, a Mamba model constrained to have only positive eigenvalues (avoiding convergence) achieved a perplexity of 16.66, compared to 17.28 for a model with negative eigenvalues.
  • Vision Tasks: Adding the token reordering refinement to MambaVision-T on ImageNet-1K improved Top-1 accuracy from 81.90% to 82.02% and significantly accelerated the convergence of the training loss.
  • Generalization: The reordering method consistently improved test perplexity across Mamba, H3, and even Transformer architectures (when converted to Mamba format).
Figure 1: The graphs of tokens (left) and the heat map of the hidden attention matrices (right) for the convergence scenario with L = 10 tokens and parameters µ = −1.58, S∆= −0.17, a = −1.08, as well as initial data x(0) = (−1.79, −0.34, 0.46, −1.25, 0.83, −0.83, 1.81, −1.16, 0.13, −0.19). In this case, all tokens and hidden attention scores tend to zero as t approaches infinity.

Limitations and Future Work

While groundbreaking, the study has a few caveats:

  • Dimensionality: The core proofs are currently restricted to tokens with a single feature channel ($D=1$). While experiments suggest these findings hold for higher dimensions, the formal proof is still a work in progress.
  • Computational Overhead: The token reordering step involves a sorting operation, which introduces a slight (though minimal) increase in training time and memory usage.
  • Optimality: The authors used a learnable affine projection for reordering, but they acknowledge that this might not be the optimal way to rank token importance.

Future research will likely focus on extending these proofs to multi-dimensional tokens and exploring more sophisticated reordering strategies to further squeeze performance out of the Mamba architecture.

Summary updated

Abstract

Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity. We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance. For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training. Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance. Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications.
BibTeX
@inproceedings{vo2025demystifying,
  author = {Thieu N Vo and Tung D. Pham and Xin T. Tong and Tan Minh Nguyen},
  title = {{Demystifying the Token Dynamics of Deep Selective State Space Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=qtTIP5Gjc5},
}
Figure