Abstract
As one of the most representative DL techniques, Transformer architecture has empowered numerous advanced models, especially the large language models (LLMs) that comprise billions of parameters, becoming a cornerstone in deep learning. Despite the impressive achievements, Transformers still face inherent limitations, particularly the time-consuming inference resulting from the quadratic computation complexity of attention calculation. Recently, a novel architecture named Mamba, drawing inspiration from classical state space models (SSMs), has emerged as a promising alternative for building foundation models, delivering comparable modeling abilities to Transformers while preserving near-linear scalability concerning sequence length. This survey conducts an in-depth investigation of recent Mamba-associated studies, covering the advancements of Mamba-based models, the techniques of adapting Mamba to diverse data, and the applications where Mamba can excel.
Introduction: The Quadratic Bottleneck
For years, the artificial intelligence field has grappled with a fundamental trade-off in sequence modeling. Transformers, while dominant, suffer from $O(L^2)$ complexity—where doubling the sequence length quadruples the computational cost. This "quadratic bottleneck" makes processing extremely long sequences, such as high-resolution images or long documents, prohibitively expensive. Conversely, Recurrent Neural Networks (RNNs) scale linearly ($O(L)$) but are difficult to train in parallel and often fail to capture long-range dependencies due to vanishing gradients.
Traditional State Space Models (SSMs) were proposed as a middle ground, offering linear scaling. However, their time-invariant nature—applying the same transformation to every token regardless of content—limited their ability to perform the complex, content-dependent reasoning required for modern AI tasks. Mamba addresses these issues by introducing a selection mechanism and hardware-aware optimizations, positioning itself as a legitimate challenger to the Transformer.
Methodology: The Evolution of Mamba
The Mamba architecture represents a significant leap in selective sequence modeling. The survey categorizes its evolution into several key pillars:
1. Mamba-1: Selection and Hardware-Awareness
The core innovation of Mamba-1 is the Selection Mechanism. Unlike previous SSMs, Mamba makes its transition matrices ($\mathbf{A, B, C}$) functions of the input. This allows the model to "choose" what to remember or ignore, effectively acting as a compressed memory. To ensure this doesn't sacrifice speed, the authors developed a Parallel Associative Scan and optimized memory recomputation, specifically targeting the GPU memory hierarchy (SRAM vs. HBM) to achieve 3x higher throughput than Transformers.
2. Mamba-2: Structured State-Space Duality (SSD)
Mamba-2 introduces the concept of SSD, a theoretical framework proving that selective SSMs and a specific form of linear attention are mathematically dual. This realization allows Mamba-2 to be 2-8x faster than its predecessor while utilizing existing Transformer optimization tricks, such as tensor parallelism.
3. Architectural Adaptations
Because Mamba is inherently 1D, researchers have developed novel scanning modes to adapt it to multi-dimensional data. These include Bidirectional Scans for text, Cross-Scans for 2D images, and Omni-Scans for 3D spatial data. Hybrid designs like Jamba also exist, interleaving Mamba blocks with standard Attention layers to balance efficiency and modeling power.
Performance and Complexity Analysis
The primary advantage of Mamba lies in its efficiency profile. Unlike Transformers, which require a massive KV-cache that grows with sequence length, Mamba maintains a fixed-size hidden state, enabling near-linear scaling.
As summarized in the table above, SSMs achieve the parallel training efficiency of Transformers while maintaining the fast, recurrent inference speed of RNNs. This makes them uniquely suited for long-context applications where Transformers become computationally unfeasible.
Key Results and Applications
Mamba-based models have shown remarkable performance across diverse domains:
- Computer Vision: Vision Mamba (Vim) is 2.8x faster than the standard Vision Transformer (DeiT) and saves 86.8% GPU memory when processing high-resolution images.
- Language Modeling: Large-scale models like Falcon Mamba 7B and Jamba (52B) demonstrate that Mamba can match or exceed the performance of state-of-the-art Transformers like Llama-3 at comparable scales.
- Specialized Domains: Mamba has been successfully applied to medical imaging (U-Mamba), 3D point cloud processing, and financial time-series analysis, where long-range dependencies are critical.
Limitations and Future Work
Despite its promise, Mamba faces several hurdles. It has yet to be tested at the "frontier" scale of 100B+ parameters. Furthermore, the ecosystem for Mamba—including tools for quantization, Low-Rank Adaptation (LoRA), and Retrieval-Augmented Generation (RAG)—is still in its infancy compared to the mature Transformer ecosystem. Future research will likely focus on scaling laws, trustworthiness, and further hardware-software co-design to solidify Mamba's place in the next generation of foundation models.
Summary updated