Visual State Space Models for Linear Complexity Global Context

463 concepts294 papers5 proposals

For years, computer vision has faced a fundamental architectural trade-off. Convolutional Neural Networks (CNNs) scale linearly ($O(N)$) with the number of pixels, making them highly efficient, but their local receptive fields struggle to capture long-range dependencies. Conversely, Vision Transformers (ViTs) capture global context via self-attention but suffer from quadratic computational and memory complexity ($O(N^2)$). This makes Transformers prohibitively expensive for high-resolution images, dense prediction tasks, and long video sequences.

Early attempts sought to achieve a global view with linear complexity by decomposing adjacency matrices, such as the spatial pyramid graph reasoning proposed by Xia Li et al. in Spatial Pyramid Based Graph Reasoning for Semantic Segmentation. However, a more generalized, robust framework emerged with the adaptation of State Space Models (SSMs)—specifically the Mamba architecture—to the visual domain. Visual State Space Models (VSSMs) bridge this gap, delivering a global receptive field with linear $O(N)$ scaling.

Adapting 1D State Spaces to 2D Spatial Data

State Space Models are fundamentally 1D causal sequence models. They process data sequentially, updating a continuous hidden state over time. To apply this to a 2D image, the spatial grid must be flattened into a sequence. However, a simple raster scan destroys 2D spatial continuity and introduces a severe directional bias.

To preserve spatial context, VSSMs employ multi-directional scanning mechanisms. Instead of a single pass, the image is unrolled and scanned from multiple perspectives simultaneously. For example, the framework in StateSpace-SSL: Linear-Time Self-supervised Learning for Plant Disease Detection utilizes directional and recurrent scanning to capture long-range spatial dependencies.

To completely eliminate directional bias, researchers have expanded the number of scanning pathways. In CountMamba: Exploring Multi-directional Selective State-Space Models for Plant Counting, the authors implement a four-direction scan (horizontal, vertical, diagonal, and anti-diagonal) to capture fine-grained plant distributions regardless of orientation. For large-scale satellite imagery, Sijie Zhao et al. introduce an Omnidirectional Selective Scan Module (OSSM) in RS-Mamba for Large Remote Sensing Image Dense Prediction, allowing the model to process high-resolution imagery as a single continuous input without orientation artifacts. By aggregating these distinct directional passes, every pixel can effectively interact with every other pixel, establishing a global receptive field at every layer, as demonstrated by Eric Nguyen et al. in S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces.

Input-Dependent Selection and Hidden Memory

Standard linear attention or traditional RNNs compress information into a fixed-size state, but they treat every token equally, leading to information bottlenecks over long sequences. Mamba overcomes this through selectivity: its discretization parameters are dynamically computed based on the input. This allows the network to decide what information to compress into the hidden state and what to discard.

In visual tasks, this selective hidden memory acts as a hardware-efficient implementation of an integral kernel. As Jianwei Zheng demonstrates in Alias-Free Mamba Neural Operator, this mathematical formulation enables global spatial dependency modeling without ever materializing an explicit $N \times N$ attention matrix. Instead of computing pairwise similarities between all pixels, the model propagates context through a compact hidden state.

This mechanism is uniquely suited for preserving spatial continuity. For instance, Soumyaroop Nandi et al. show in BioTamperNet: Affinity-Guided State-Space Model Detecting Tampered Biomedical Images that the sequence-modeling strengths of linear SSMs preserve spatial continuity across wide fields of view, making it easier to distinguish natural organic textures from artificial duplications in image forensics.

Unlocking High-Resolution and Dense Prediction

The true power of $O(N)$ global modeling becomes apparent in tasks where downsampling is unacceptable because fine details are critical.

Image Restoration and Super-Resolution

When upscaling images by large factors (e.g., $\times 4$ or $\times 16$), models must maintain global structural context to avoid spatial fragmentation. In the NTIRE 2025 challenge documented by Zheng Chen et al. (NTIRE 2025 Challenge on Image Super-Resolution (x4): Methods and Results), SSMs emerged as a highly efficient alternative to Transformers for capturing long-range spatial dependencies. Similarly, the authors of Thermal Image Super-Resolution Challenge Results - PBVS 2025 found that integrating SSMs allows for efficient processing of high-resolution outputs while maintaining global context. Yongsong Huang et al. utilize this in IRSRMamba: Infrared Image Super-Resolution via Mamba-based Wavelet Transform Feature Modulation Model to achieve superior peak signal-to-noise ratios without memory constraints.

Medical Imaging and Dense Segmentation

High-resolution medical scans require precise localization alongside global anatomical context. Wenliang Tang demonstrates in VM-UNet++ research on crack image segmentation based on improved VM-UNet that selective scanning within visual state space blocks provides a global receptive field with the efficiency typically associated with convolutional layers. In precision ophthalmology, Shun Zou et al. show in OCTAMamba: A State-Space Model Approach for Precision OCTA Vasculature Segmentation that integrating selective SSMs into a U-shaped architecture reduces parameter counts up to twelvefold without sacrificing segmentation accuracy.

Remote Sensing and Multimodal Fusion

Satellite imagery involves massive pixel grids where broad atmospheric patterns must be modeled alongside fine ground details. Siran Peng shows in FusionMamba: Efficient Remote Sensing Image Fusion With State Space Model that state space models offer a scalable alternative to attention-based architectures, allowing models to capture global context across large satellite images without prohibitive memory overhead.

Boundaries and Open Questions

While VSSMs offer a compelling alternative to Transformers, recent work has begun to map their limitations. A key question is whether the complexity of an SSM is always justified. Weihao Yu et al. address this in MambaOut: Do We Really Need Mamba for Vision?, demonstrating that while Mamba's recurrent memory is overkill for simple image classification, it remains a legitimate superpower for high-resolution tasks like detection and segmentation where sequence lengths explode.

Robustness is another critical frontier. Hashmat Shadab Malik et al. conducted a comprehensive stress test in Towards Evaluating the Robustness of Visual State Space Models, revealing that while VSSMs handle occlusions and common corruptions exceptionally well, they exhibit a unique sensitivity to high-frequency adversarial attacks.

Finally, the field continues to explore alternative linear-complexity paradigms. For example, Zhenkai Wu et al. evaluate the Extended Long Short-Term Memory architecture in CDxLSTM: Boosting Remote Sensing Change Detection With Extended Long Short-Term Memory, while the authors of Towards Remote Sensing Change Detection with Neural Memory introduce persistent neural memory to achieve similar linear-scaling global context.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 296 papers
  2. Version 1 · 2026-10-04 · 457 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_complexity_global_modeling_bb431fdb,
  title = {linear-complexity global modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-complexity-global-modeling/rdr_bb431fdba083463b841c0fd7ef7e12e1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_linear_complexity_global_modeling_bb431fdb_v2,
  title = {linear-complexity global modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-complexity-global-modeling/rdr_bb431fdba083463b841c0fd7ef7e12e1/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake