The dominant architectures in modern deep learning—Transformers and State Space Models (SSMs) like Mamba—were originally forged for language. Language is inherently one-dimensional: words follow words in a strict linear sequence.
When we adapt these models to computer vision, we encounter a fundamental structural mismatch. Images and videos are multi-dimensional grids rich in geometric topology. The standard bridge between these two worlds is linearization: flattening a 2D image or a 3D video volume into a 1D sequence of patches or pixels.
However, this flattening process fundamentally disrupts spatial locality. When you force a multi-dimensional grid into a single line, pixels that are physically adjacent in the original space are violently torn apart in the sequence. This spatial-locality gap introduces severe representation artifacts, forcing models to work redundantly to relearn basic geometric relationships.
The Geometry of the Flattening Artifact
To understand why standard flattening fails, consider a simple $8 \times 8$ pixel grid processed via a traditional row-by-row raster scan.
Raster Scan Discontinuity:
[ Row 1: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 ]
↴ (Massive Jump)
[ Row 2: 9 → 10 → ... ]
This naive mapping introduces two severe geometric distortions:
- The Row-Jump Discontinuity: Pixel 8 (the right edge of the first row) and Pixel 9 (the left edge of the second row) become immediate neighbors in the 1D sequence. In the actual image, they sit on opposite sides of the canvas. As Yang et al. observe, these artificial boundaries disrupt the local semantic continuity that sequence models rely on to maintain a coherent hidden state.
- The Orthogonal Blind Spot: Consider Pixel 2 and Pixel 10. In the 2D image, Pixel 10 sits directly beneath Pixel 2—they are highly correlated vertical neighbors. In the 1D sequence, however, they are separated by the entire width of the image. Because sequence models experience signal decay over long token distances, this artificial separation makes it incredibly difficult to capture relationships along the axis orthogonal to the scan direction, creating severe directional blind spots.
The Cost of Disruption
When spatial topology is discarded, downstream performance suffers across a variety of visual domains:
- Object Hallucination: In Large Vision-Language Models (LVLMs), raster scanning creates an artificial distance between vertically adjacent patches. Yun Xing argues that this linear representation fails to preserve natural spatial topology, directly contributing to object hallucinations where models confidently describe non-existent objects. Li et al. similarly show that this structural gap prevents the model from maintaining coherent spatial relationships during visual text generation.
- Abstract Reasoning Failures: On benchmarks like the Abstraction and Reasoning Corpus (ARC), which require precise grid-based transformations, standard Vision Transformers fail spectacularly. Li et al. demonstrate that flattening images into 1D sequences destroys the exact topological information and explicit grid boundaries required to solve abstract geometric puzzles.
- Loss of Fine-Grained Context: For tasks requiring high-fidelity localization—such as medical image segmentation or infrared small target detection—local contrast and precise boundaries are everything. Yan et al. note that spatial discontinuities make it difficult for sequence models to capture complex anatomical shapes. Zhang et al. show that destroying local neighborhood similarity prevents models from recognizing the fine-grained local contrast patterns essential for identifying tiny anomalies.
Redesigning the Scan Path
If the core issue is the trajectory of the 1D path, can we design a scan that respects 2D space? Researchers have proposed several advanced space-filling curves to mitigate flattening artifacts.
Continuous and S-Shaped Scans
To eliminate the massive coordinate jumps inherent to raster scanning, Yang et al. implement a continuous "zigzag" scanning process. By reversing direction at the end of each row, the scan ensures that the last token of row $N$ remains adjacent to the first token of row $N+1$. Similarly, Li et al. utilize a "Nested S-shaped Scanning" strategy to preserve local textures and continuity for image restoration tasks.
Fractal and Hilbert Curves
While zigzag scans fix the row-jump issue, they still suffer from the orthogonal blind spot. To keep both horizontal and vertical neighbors close in 1D space, researchers leverage fractal space-filling curves, most notably the Hilbert curve.
Hilbert Curve Scan (Local Clustering):
1 → 2 7 → 8
↓ ↑ ↓
4 ← 3 6 ← 5
A Hilbert curve folds in on itself recursively, fully exploring a local 2D quadrant before stepping into the next. Bai et al. demonstrate that replacing raster scans with Hilbert curves better preserves local spatial relationships, significantly boosting the robustness of generative defenses against adversarial attacks. In the video domain, Wu et al. extend this concept to a 3D Hilbert scanning mechanism, successfully preserving both spatial and temporal proximity across video frames to prevent "locality forgetting."
However, mathematical topology dictates that a single 1D curve can never perfectly map a 2D space without some disruption. As noted in the context of FractalMamba, some neighboring pixels will inevitably be separated. To counteract this, frameworks like TS-Mamba employ coordinate shifting ("Scan-Shift-Scan") to recover the local adjacencies that the primary scanning path misses.
Content-Aware and Dynamic Topologies
Rather than forcing an image into a rigid, predefined geometric path, a more radical approach lets the visual content itself dictate the sequence topology.
Tree-Structured Propagation
Instead of scanning a grid linearly, why not route information along the natural semantic boundaries of the objects in the scene? Xiao et al. introduce MambaTree, which dynamically constructs a Minimum Spanning Tree (MST) based on input features. Information propagates along semantic connections, ensuring that pixels belonging to the same object remain close in the processing sequence regardless of their grid coordinates.
Fixed Grid Scan vs. Dynamic Tree Scan:
Grid: [1 → 2 → 3 → 4] (Rigid, breaks object boundaries)
Tree: (Object Center)
↓
[Local Patch]
↙ ↘
[Edge Patch] [Edge Patch] (Follows semantic shape)
For multi-scale structural awareness, Xie et al. propose QuadMamba, which learns a quadtree-based selective scan. It dynamically "zooms in" on highly informative, dense regions of an image with fine-grained scans while processing uniform background regions with coarse, large-scale steps, preserving structural integrity where it matters most.
Optimization Paths
Kun Zhou reframes the serialization problem entirely by casting local pixel aggregation as a Traveling Salesman Problem (TSP). By calculating the "shortest path" across an image based on pixel visual similarity, TSP-Mamba ensures that the sequence follows the contours of visual features, completely bypassing the local pixel forgetting that plagues rigid scanning trajectories.
Multi-Axis and Windowed Inductive Biases
A final class of solutions injects 2D structural awareness directly into the model's internal mechanics, modifying how attention or state updates are calculated.
Multi-Axis Scanning
If a single scan direction creates a blind spot, the intuitive fix is to scan from multiple directions simultaneously. Frameworks like 2D-CrossScan Mamba implement multi-path information propagation to ensure spatial consistency across both axes. To do this efficiently across scales, Yuheng Shi demonstrates that a "hierarchy within a hierarchy" structure—running 2D scans at full resolution in some directions and on downsampled maps in others—allows the model to capture both immediate local context and long-range dependencies without a massive computational penalty.
Windowing and 2D Positional Encodings
Instead of serializing the entire image globally, models can restrict sequence processing to localized 2D neighborhoods. Huang et al. and Wu et al. utilize windowed selective scanning to force the model to prioritize local 2D dependencies before aggregating global context.
Finally, to address the fact that standard 1D positional encodings suffer from remote decay (where the mathematical distance between tokens causes the model to forget that patch 1 and patch 100 are vertically aligned), Liu et al. introduce 2D-aware positional encodings (RoPE-DHR). This ensures that even if patches are separated during serialization, their true geometric coordinate relationships remain preserved in the model's mathematical foundation.