The dominant paradigm in sequence modeling, the Transformer, relies on self-attention to capture dependencies across inputs. While highly expressive, self-attention requires computing pairwise interactions between every single token in a sequence. This creates a quadratic computational bottleneck: doubling the sequence length quadruples the time and memory required. For long contexts—such as high-resolution medical images, extended audio streams, or multi-year electronic health records—this quadratic scaling becomes prohibitively expensive.
To overcome this limitation, a rapidly growing body of research explores Selective State Space Models (SSMs), most notably the Mamba architecture. As surveyed by Xiao Liu et al. in Vision Mamba: A Comprehensive Survey and Taxonomy, these models replace the quadratic attention mechanism with an alternative that scales linearly ($O(N)$) with sequence length, offering a highly efficient path forward for long-range dependency modeling.
The Intuition: From Attention Matrices to Hidden States
To understand why Selective SSMs work, it is helpful to contrast how they handle memory compared to Transformers.
In a Transformer, the model maintains an explicit cache of all past tokens (the Key-Value cache). To process a new token, the model must look back at every historical token individually. This is why the computational cost grows with the square of the sequence length.
An SSM, by contrast, compresses historical information into a fixed-size internal hidden state. It operates like a continuous-time physical system or an elegant recurrence: as new data arrives, the current hidden state is updated, and the old state is partially forgotten. Because the size of this hidden state remains constant regardless of how many tokens have passed, the computational cost per token is fixed. Processing a sequence of length $N$ takes linear time ($O(N)$).
Historically, simple recurrent models or linear time-invariant (LTI) state space models struggled to match the expressive power of Transformers. Because their state transition rules were fixed and independent of the input content, they could not selectively focus on important tokens or ignore irrelevant noise.
Mamba introduces selectivity into this formulation. By making the state transition matrices functions of the current input token, the model dynamically adjusts its filtering behavior. It can choose to compress and remember critical context while flushing out irrelevant details. To maintain training efficiency, this input-dependent recurrence is computed using a hardware-aware parallel scan algorithm on the GPU, bypassing the sequential bottleneck traditionally associated with recurrent architectures.
Validating Expressivity and In-Context Learning
A natural question arises: does compressing history into a fixed-size hidden state sacrifice the sophisticated reasoning capabilities of Transformers?
Recent evaluations suggest it does not. For instance, Riccardo Grazzi et al. demonstrate in Is Mamba Capable of In-Context Learning? that Mamba achieves in-context learning capabilities comparable to Transformers on both simple function approximations and complex natural language processing tasks. This indicates that a selective hidden state can store and manipulate abstract task representations just as effectively as an explicit attention matrix.
Furthermore, the success of SSMs has inspired alternative linear-time paradigms. Jiqun Chu et al. show in Incorporating Exponential Smoothing into MLP: a Simple but Effective Sequence Model that integrating classical exponential smoothing (ETS) with learnable parameters into a standard Multi-Layer Perceptron can yield a highly efficient sequence model (ETSMLP) that performs competitively with state-of-the-art SSMs on long-range benchmarks.
Adapting 1D Recurrences to Multi-Dimensional Data
Because SSMs are fundamentally designed for 1D sequential data, adapting them to multi-dimensional domains like computer vision requires creative structural mappings. To process an image or a volume, researchers flatten the data into a sequence, but the choice of how to traverse the pixels or voxels matters immensely for preserving spatial context.
Spatial Scanning Strategies
To capture continuous structures effectively, models employ specialized scanning paths:
- Depth-First Scanning: Shengli Chen et al. introduce a depth-first scanning mechanism in Mono-ViM to optimize monocular depth estimation in endoscopic videos, ensuring the model tracks continuous anatomical structures.
- Pyramidal Scanning: Nasar Iqbal et al. propose a multi-scale pyramidal scanning strategy in Pyramid-based Mamba Multi-class Unsupervised Anomaly Detection to detect and localize minute visual anomalies across varying spatial resolutions.
High-Resolution Medical Imaging
Medical imaging is a prime beneficiary of linear scaling, as 3D volumes and multi-modal scans quickly overwhelm Transformer-based architectures.
- Segmentation: Ju Wang et al. combine the localized feature extraction of CNNs with the global, linear-time sequence modeling of Mamba in Cascade Residual Multiscale Convolution and Mamba-Structured UNet for Advanced Brain Tumor Image Segmentation.
- Reconstruction and Translation: Jiahao Huang et al. develop Enhancing Global Sensitivity and Uncertainty Quantification in Medical Image Reconstruction with Monte Carlo Arbitrary-Masked Mamba (MambaMIR) to achieve global context sensitivity without quadratic overhead. Similarly, Jing Zou et al. introduce MMR-Mamba for multi-modal MRI reconstruction, while Chaohui Gong et al. leverage Mamba's receptive field in MM2CT for MR-to-CT image translation.
Advanced Vision Domains
Beyond standard imagery, the linear-time complexity of SSMs enables real-time processing of complex visual data. The authors of DMS2F-HAD utilize dual-branch Mamba blocks to extract spatial and spectral features simultaneously for hyperspectral anomaly detection. In low-light image enhancement, the authors of ExpoMamba decouple amplitude and phase components in the frequency domain using frequency-aware SSMs, achieving substantial speedups over self-attention baselines.
A Broadening Ecosystem
The linear scaling property of selective state space models has triggered widespread adoption across diverse domains where long sequences or real-time constraints are paramount:
- Generative Modeling: Integrating Mamba into diffusion frameworks allows models to handle high-dimensional denoising tasks efficiently. Md Atik Ahamed et al. implement a Mamba-based denoiser in RefiDiff for missing data imputation, while the authors of End-to-End Multi-Modal Diffusion Mamba (MDM) demonstrate rapid inference speeds and lower latency for long-sequence image generation.
- Speech and Audio Processing: For streaming and offline speech tasks, Takafumi Moriya et al. introduce Attention-Free Dual-Mode ASR with Latency-Controlled Selective State Spaces (LC-BiMamba), replacing traditional attention with convolutional Mamba blocks. For audio restoration, Doyeop Kwak et al. design EDNet, a speech enhancement framework that leverages adaptive gating Mamba mechanisms to process extended audio sequences efficiently.
- Structured and Spatiotemporal Data: In healthcare, Mahshid Alinoori et al. develop EHRMamba to ingest massive, longitudinal patient histories within electronic health records. In text analysis, Adamu Lawan et al. utilize Mamba blocks to bridge distant aspects and opinion words in aspect-based sentiment analysis. For real-time tracking and robotics, Long Lei et al. deploy Mamba in MambaXCTrack for ultrasound needle tracking, and Wenhao Yang et al. integrate it into Spatiotemporal Contextual 3D Semantic Segmentation for Intelligent Outdoor Mining to process historical point cloud sequences.
By compressing history into a dynamic, selective hidden state rather than computing exhaustive pairwise comparisons, Selective SSMs break the quadratic chains of attention. They offer a mathematically elegant and computationally viable path toward processing the massive, continuous data streams of the real world.
Key Foundational Work
Cascade Residual Multiscale Convolution and Mamba-Structured UNet for Advanced Brain Tumor Image Segmentation (2024) — contributed 2 concept elements to this direction.
Is Mamba Capable of In-Context Learning? (2024) — contributed 1 concept elements to this direction.
Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2024) — contributed 1 concept elements to this direction.