Selective State Space Models (SSMs), particularly the Mamba architecture, have taken the AI world by storm by offering Transformer-like performance with linear scaling. However, while they work exceptionally well in practice, we haven't fully understood why from a theoretical standpoint.
In a new paper accepted at ICLR 2025, researchers from the National University of Singapore and FPT Software AI Center pull back the curtain on Mamba’s inner workings. By treating the model as a continuous-time dynamical system, they reveal how tokens evolve as they pass through deep layers, leading to practical refinements that boost performance in both language and vision tasks.
TL;DR
This paper derives the continuous-time limit of the Mamba model to prove that tokens either vanish to zero or explode to infinity as they go deeper. By ensuring tokens diverge and reordering them based on a novel "importance score," the authors significantly improve Mamba's accuracy and training speed.
What Problem Does This Paper Address?
Despite their success, deep selective SSMs remain somewhat of a "black box." Specifically, we lack a comprehensive theoretical understanding of how tokens—the individual units of data like words or image patches—change as they move through the model's layers.
In Transformers, token dynamics are known to exhibit exponential divergence. In Mamba, the "selective" nature (where parameters change based on the input) makes the math much harder. The authors sought to answer: What is the long-term fate of a token in a deep Mamba model, and how does that fate affect the model's ability to learn?
Methods
The researchers approached Mamba through the lens of Neural ODEs. They derived a continuous-time dynamical system that represents the Mamba model as the number of layers goes to infinity.
Key technical steps included:
- Continuous-Time Derivation: They simplified the S6 layer (Mamba's core) into a system of differential equations.
- Asymptotic Analysis: They analyzed the stability of these equations to see if solutions (tokens) converge or diverge over "time" (model depth).
- Eigenvalue Analysis: They identified that the symmetric part of the input-output matrix ($S_C^\top S_B$) acts as a "switch" for the model's behavior.
- Token Reordering: Based on their findings that tokens diverge at different rates, they proposed a SoftSort-based reordering mechanism. This ranks tokens by an "importance score" before they enter the S6 layer, ensuring the most critical data is prioritized during training.
Datasets
The authors validated their theoretical findings across several standard benchmarks:
- Language Modeling: WikiText-103 (using a 130M parameter Mamba model).
- Image Classification: ImageNet-1K (using the MambaVision-T baseline).
- Text Classification: IMDB and AGNews.
Key Insights and Ideas
- The Binary Fate of Tokens: In the one-dimensional case, tokens follow an "all or nothing" rule. Depending on the model parameters, either all tokens converge to zero (vanishing) or all tokens diverge to infinity (exploding).
- Convergence is a Performance Killer: When tokens converge to zero, the model loses diversity and randomness, which the authors empirically proved leads to higher perplexity (worse performance) in language tasks.
- Unequal Contribution: In the divergence scenario, tokens don't explode at the same speed. Tokens with higher initial values diverge faster. This means the model naturally "prioritizes" certain tokens over others during the update process.
- The Importance Switch: The sign of the eigenvalues of the input-output matrix determines the scenario. Positive eigenvalues lead to the beneficial divergence scenario, while negative ones lead to the harmful convergence scenario.
Key Results
The theoretical refinements led to measurable gains:
- Language Modeling: On WikiText-103, a Mamba model constrained to have only positive eigenvalues (avoiding convergence) achieved a perplexity of 16.66, compared to 17.28 for a model with negative eigenvalues.
- Vision Tasks: Adding the token reordering refinement to MambaVision-T on ImageNet-1K improved Top-1 accuracy from 81.90% to 82.02% and significantly accelerated the convergence of the training loss.
- Generalization: The reordering method consistently improved test perplexity across Mamba, H3, and even Transformer architectures (when converted to Mamba format).
Limitations and Future Work
While groundbreaking, the study has a few caveats:
- Dimensionality: The core proofs are currently restricted to tokens with a single feature channel ($D=1$). While experiments suggest these findings hold for higher dimensions, the formal proof is still a work in progress.
- Computational Overhead: The token reordering step involves a sorting operation, which introduces a slight (though minimal) increase in training time and memory usage.
- Optimality: The authors used a learnable affine projection for reordering, but they acknowledge that this might not be the optimal way to rank token importance.
Future research will likely focus on extending these proofs to multi-dimensional tokens and exploring more sophisticated reordering strategies to further squeeze performance out of the Mamba architecture.
Summary updated