State-Space Models (SSMs) like Mamba, alongside traditional recurrent architectures like LSTMs, offer an alluring promise for computer vision: the ability to process data with linear computational complexity relative to sequence length. However, these architectures are fundamentally one-dimensional. They ingest tokens sequentially, updating a hidden state step-by-step.
To process a two-dimensional image or a three-dimensional video through a 1D sequence model, the data must first be flattened—a process known as serialization. The choice of how we map multi-dimensional space into a linear stream dictates how effectively a model can preserve spatial context.
The Spatial Locality Gap
The traditional default for flattening images is raster scanning (row-by-row translation). While simple, raster scanning fundamentally breaks 2D topology.
Consider an image of width $W$. Two pixels vertically adjacent to one another—at coordinates $(x, y)$ and $(x, y+1)$—are separated by a spatial distance of exactly 1 pixel in 2D space. However, after a raster scan, the first pixel appears at index $k$, while the second is pushed to index $k + W$.
Raster Scan:
[ P1 -> P2 -> P3 -> P4 ] --> Breaks continuity at the row edge!
[ P5 -> P6 -> P7 -> P8 ] --> P4 and P8 are far apart in 1D sequence.
This spatial separation introduces a severe bottleneck for sequence models. Because architectures like SSMs and LSTMs exhibit a recency bias or finite effective memory windows, forcing vertically adjacent tokens apart causes "locality forgetting" or "historical decay." The model's hidden state loses relevance during large spatial jumps, degrading its ability to capture fine-grained local structures.
Continuous and Zigzag Scanning
The most immediate fix for the spatial-locality gap is eliminating the abrupt jump from the end of one row to the beginning of the next. By reversing the direction of every alternating row, we create a continuous, snake-like path.
Zigzag / Snake Scan:
[ P1 -> P2 -> P3 -> P4 ]
| --> Continuous transition preserves boundary proximity.
[ P8 <- P7 <- P6 <- P5 ]
Hu et al. introduce this concept to diffusion backbones in ZigMa: A DiT-style Zigzag Mamba Diffusion Model. By ensuring that scanning paths maintain proximity between neighboring pixels, the zigzag pattern preserves spatial continuity and significantly improves generative quality compared to standard row-wise sweeps.
Similarly, Yang et al. leverage continuous zigzag scanning in PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition to ensure that tokens adjacent in the sequence remain physically adjacent in 2D space, preventing the hidden state from losing semantic relevance.
This continuity is highly beneficial across diverse vision tasks. In image style transfer, Liu et al. implement continuous scanning paths in SaMam: Style-aware State Space Model for Arbitrary Image Style Transfer to achieve more coherent global feature extraction. For dense pixel-level tasks like low-dose CT denoising, Yang et al. show in CT-Mamba that Z-shaped continuous scanning prevents the loss of local structural information at image boundaries.
Mathematical Locality via Space-Filling Curves
While zigzag scanning solves the row-end discontinuity, it still prioritizes one axis over another, leaving vertical relationships weaker than horizontal ones. To preserve multi-directional proximity, researchers utilize fractal space-filling curves, such as the Hilbert curve and the Morton (Z-order) curve.
A Hilbert curve folds recursively through 2D space, ensuring that the 1D distance between any two serialized points closely bounds their true 2D geometric distance. This clustering property means that a local neighborhood in an image remains a local neighborhood within the 1D sequence.
Hilbert Curve (Fractal Quadrant Clustering):
[ P1 P2 ]--[ P15 P16 ]
| | | |
[ P4 -- P3 ] [ P14 P13 ]
| |
[ P5 -- P6 ] [ P11 --P12 ]
| |
[ P8 -- P7 ]--[ P10 P9 ]
Xiao et al. demonstrate the power of this approach in Boosting Vision State Space Model with Fractal Scanning. By replacing linear scans with Hilbert curves, their architecture preserves structural integrity across varying image resolutions without incurring quadratic computational costs.
This mathematical locality has been applied to several domains:
- Video Processing: Jeong et al. utilize Hilbert curves in LC-Mamba: Local and Continuous Mamba with Shifted Windows for Frame Interpolation to maintain spatial correlations and mitigate historical decay.
- Anomaly Detection: He et al. employ Hilbert curves in MambaAD to map 2D clusters into 1D sequences, allowing the model to isolate localized structural anomalies.
- Adverse Weather Recovery: Wang et al. use Morton-order scanning in MODEM to separate global weather patterns from localized degradation artifacts like raindrops or snowflakes.
Mitigating Curve Junctions with Shifted Grids
Despite their mathematical elegance, static space-filling curves possess an inherent flaw: junction blind spots. Two pixels can be perfectly adjacent in 2D space but sit on opposite sides of a major fractal boundary, causing them to be separated by a massive distance along the 1D path.
To resolve this, architectures introduce shifted scanning grids. Xiao et al. apply a coordinate shift to the scanning curve across different layers, ensuring that diagonal or non-adjacent neighbors on the primary curve are processed in close proximity in subsequent layers.
Similarly, in video super-resolution, Zhu et al. develop a "Scan-Shift-Scan" mechanism in Trajectory-aware Shifted State Space Models for Online Video Super-Resolution. By alternating the scanning grid between layers, pixels located at the boundaries of one sequence are centered in another, mitigating the artifacts typically caused by 1D linearization.
Task-Specific and Content-Adaptive Trajectories
Fixed geometric curves assume that every pixel relationship is uniform. However, semantic information in images is rarely distributed evenly. The latest evolution in locality-preserving mapping focuses on trajectories that adapt to the underlying data or task geometry.
Manhattan Distance Alignment
Instead of forcing a curve to fit a grid, Zhao et al. propose Polyline Path Masked Attention for Vision Transformer. They utilize L-shaped polyline paths to connect tokens such that the sequence distance between any two points directly corresponds to their Manhattan distance in the 2D grid, explicitly preserving spatial adjacency relationships.
Concentric and Spiral Paths
For tasks where the center of the image contains the most vital context, rigid grid curves can be suboptimal. In Large Vision-Language Models (LVLMs), Xing et al. discover that traditional row-by-row flattening contributes to object hallucination due to attention decay over long sequences.
In Mitigating Object Hallucination via Concentric Causal Attention, the authors organize image tokens into an inward-spiraling, concentric sequence. This structural alignment minimizes the maximum distance between any visual element and the central text instruction, maintaining tighter focus across the image.
Concentric / Spiral Scan:
[ P1 -> P2 -> P3 -> P4 ]
[ P12 -> P13 -> P14 P5 ] --> Centers the focus inward,
[ P11 P16 <- P15 P6 ] minimizing attention decay
[ P10 <- P9 <- P8 <- P7 ] to the core semantic region.
Content-Adaptive Routing
The most flexible approach lets the image content define its own scanning path. In TSP-Mamba: The Travelling Salesman Problem Meets Mamba for Image Super-resolution and Beyond, Zhou et al. treat pixel serialization as a Traveling Salesman Problem.
Instead of following a fixed geometric path, the model calculates a trajectory based on pixel similarity. The scan dynamically follows edges, textures, and continuous semantic structures. By aggregating highly correlated local neighborhoods sequentially, the model prevents the "forgetting" of structural details during lightweight image super-resolution.
Content-Adaptive (TSP) Scan:
[ Edge1 -> Edge2 ] [ Texture1 ]
\ /
[ Edge3 ]--[ Edge4 -- Texture2 ] --> Path follows semantic features
rather than rigid grid lines.
By tailoring the serialization path to respect both the mathematical constraints of 2D space and the semantic boundaries of visual features, these techniques allow linear-time sequence models to retain the spatial inductive biases that make traditional convolutions and 2D attention mechanisms so powerful.
Key Foundational Work
Pixel-Oriented Visualization Techniques for Exploring Very Large Data Bases (1996) — contributed 1 concept elements to this direction.