The rise of the Vision Transformer (ViT) established self-attention as the standard for capturing global context in visual tasks. However, self-attention suffers from a fundamental scalability bottleneck: its computational and memory complexities scale quadratically, $\mathcal{O}(N^2)$, with the number of tokens $N$. When an image's resolution scales up—such as in 4K image restoration, medical imaging, or high-resolution remote sensing—the number of tokens explodes, leading to memory exhaustion and prohibitive computational costs.
To break this bottleneck, a new paradigm adapts linear-complexity sequence models, such as State Space Models (SSMs) and linear attention variants, to 2D spatial data. By achieving $\mathcal{O}(N)$ scaling, these architectures process high-resolution images globally without the quadratic overhead of traditional transformers.
The Core Challenge: Causality vs. Spatiality
The primary obstacle to using models like Mamba or RWKV for vision is that they were inherently designed for 1D causal sequences, such as text or time-series data. In a 1D causal model, information flows strictly from the past to the future. The model updates a compressed hidden state vector sequentially:
$$h_t = \bar{A} h_{t-1} + \bar{B} x_t$$
This recurrence allows the hidden state $h_t$ to act as a running summary of the sequence up to step $t$, requiring only constant time to ingest each new token.
Images, however, are non-causal and multi-dimensional. A pixel's semantic meaning depends heavily on its neighbors in all directions—up, down, left, and right. If you simply flatten a 2D image into a 1D sequence using a standard raster scan, pixels that are vertically adjacent in space become separated by massive distances in the sequence. Furthermore, a standard causal pass prevents pixels early in the sequence from receiving context from pixels later in the sequence.
Bridging the Gap with Multi-Directional Scanning
To capture 2D spatial dependencies using 1D linear operations, researchers deploy specialized scanning mechanisms. Instead of relying on a single sequence pass, the image is unrolled into multiple sequences along different geometric paths. Each path is processed by its own linear model instance, and the resulting hidden states are combined to form a comprehensive spatial representation.
Standard Raster Scan: Multi-Directional Scan (e.g., 4-Way):
┌───►───►───┐ ┌───►───►───┐ ┌───◄───◄───┐
│ ┌─►───►─┐ │ │ │ │ │
│ │ │ │ ▼ ▲ ▼ ▲ ▼ ▲
└─┴───────┴─┘ │ │ │ │
(Breaks vertical └───►───►───┘ └───◄───◄───┘
continuity) (Combines horizontal & vertical paths)
By running these directional scans simultaneously, the network ensures that every pixel aggregates context from its entire surroundings. At any given coordinate, the combined hidden states contain information propagated from different edges of the image, effectively simulating a global receptive field at a fraction of the cost.
Advanced Scanning Topologies
Different architectures employ distinct scanning strategies to optimize the balance between spatial coverage and computational speed:
- Omnidirectional and Multi-Way Scans: To handle the orientation-agnostic nature of complex imagery, models expand the number of scanning paths. For instance, Zhao et al. introduce an Omnidirectional Selective Scan Module (OSSM) for remote sensing data, allowing features to be aggregated across four orthogonal directions. For low-level image restoration, Shi et al. propose an "Omni Selective Scan" that tracks dependencies across six distinct directions, ensuring fine structural details are preserved.
- Continuous Paths: Standard raster scans introduce artificial boundaries when jumping from the end of one row to the beginning of the next. To preserve physical proximity in the token sequence, Yang et al. design a continuous "zigzag" scanning process in PlainMamba, ensuring that tokens adjacent in the 1D sequence remain spatial neighbors in 2D space.
- Direction Flipping and Global Tokens: Multi-directional scanning can introduce its own overhead due to the creation of multiple parallel sequences. To streamline this, Wang et al. demonstrate with the Adventurer architecture that complex multi-directional scans can be bypassed. Instead, they alternate the sequence direction between successive network layers and utilize a global "heading average" token to distribute context across the entire image efficiently.
Beyond SSMs: Linear 2D Attention
The shift toward linear complexity is not exclusive to State Space Models; it also encompasses linear attention frameworks like RWKV. Standard attention computes similarity between all pairs of queries ($Q$) and keys ($K$) before multiplying by values ($V$), resulting in $(QK^T)V$ with $\mathcal{O}(N^2)$ complexity. Linear attention changes the order of operations to $Q(K^TV)$, reducing the complexity to $\mathcal{O}(N)$ by avoiding the explicit construction of the massive $N \times N$ attention matrix.
To adapt this to vision, Chai et al. utilize bidirectional processing and frequency-domain analysis to enhance image denoising performance. Similarly, Li et al. introduce a 2D-WKV attention mechanism combined with a spatial shift module, enabling isotropic spatial reasoning for remote sensing tasks without the computational burden of standard transformers.
Empirical Impact and Efficiency Gains
The practical advantages of linear-complexity spatial modeling become starkly apparent when processing large-scale or high-resolution datasets. By avoiding the memory explosion of self-attention, these models unlock capabilities that were previously mathematically or hardware-prohibitive.
High-Resolution Scaling
For massive inputs, the difference in scaling laws translates to drastic reductions in floating-point operations. Yang et al. note that when processing $4096 \times 4096$ images, a linear state-space approach achieves over a twenty-fold reduction in computational cost compared to traditional vision transformers. This efficiency allows models to process high-resolution images as single, unified inputs rather than breaking them into isolated patches, preventing information loss across artificial boundaries.
Resource-Constrained Deployment
In dense prediction and real-time detection tasks, linear spatial models drastically lower the hardware barrier to entry. For example, Chen et al. demonstrate that a nested Mamba architecture for infrared small target detection runs 8× faster and consumes 62.2% less GPU memory than top-tier transformer models. Similarly, Wang et al. integrate an SSM backbone into the YOLO framework, yielding significant accuracy improvements for real-time industrial and medical object detection while maintaining a highly efficient memory footprint.
Low-Level Vision and Super-Resolution
In image restoration, where preserving pixel-level fidelity across large dimensions is critical, linear models offer a powerful alternative to local-window transformers. As shown in benchmarks discussed by Guo et al. and Adhikarla et al., incorporating local enhancements with linear state-space blocks allows models to maintain consistent inference speeds across varying resolutions up to 4K.
Furthermore, Rivadeneira highlights that in extreme upscaling scenarios—such as $\times 8$ and $\times 16$ thermal image super-resolution—Mamba-based models successfully maintain high reconstruction quality where standard attention mechanisms fail due to memory limits. This scalability is echoed across broader domain surveys, such as the remote sensing review by Qi et al., which underscores SSMs as a foundational shift for handling massive pixel volumes efficiently.
Key Foundational Work
Plant Disease Detection Using Self-Supervised Learning: A Systematic Review (2024) — contributed 2 concept elements to this direction.
NTIRE 2024 Challenge on Low Light Image Enhancement: Methods and Results (2024) — contributed 2 concept elements to this direction.
NTIRE 2022 Challenge on Stereo Image Super-Resolution: Methods and Results (2022) — contributed 2 concept elements to this direction.