The rise of linear-complexity sequence models—such as State Space Models (SSMs) like Mamba, Gated Linear Attention (GLA), and advanced recurrent architectures like xLSTM and RWKV—has opened a new frontier in deep learning. These models offer the holy grail of long-range context modeling: a global receptive field with computational complexity that scales linearly ($O(N)$) rather than quadratically ($O(N^2)$) with sequence length.
However, these architectures inherit a fundamental constraint from their natural language origins: they are inherently one-dimensional and causal. They process tokens sequentially, assuming a strict "arrow of time."
When applied to spatial data like images, medical volumes, or point clouds, this 1D causal assumption breaks down. Visual data is non-causal, multi-dimensional, and isotropic—a pixel depends just as much on its neighbors below and to the right as it does on those above and to the left. To bridge this gap, researchers have developed multi-directional spatial scanning, a family of techniques that unroll multi-dimensional grids into 1D sequences along multiple geometric paths to restore spatial inductive bias without sacrificing linear efficiency.
The Problem: Spatial Discontinuity and Directional Bias
The naive way to adapt a 1D sequence model to an image is to flatten the image patches into a single string of tokens via a standard raster scan (left-to-right, top-to-bottom).
Naive Raster Scan:
[1] → [2] → [3]
↓
[4] → [5] → [6]
↓
[7] → [8] → [9]
This approach introduces two severe degradation mechanisms:
- Spatial Discontinuity: As noted by Yan Yan et al., flattening forces pixels that are vertically adjacent (such as
[3]and[6]) to be separated by a wide distance in the 1D sequence. The model's recurrent state must remember information over long token distances just to understand a local vertical edge. - Directional/Causal Bias: A standard causal sequence model updates its hidden state using only past tokens. In the diagram above, token
[5]can absorb context from[1],[2],[3], and[4], but it is completely blind to[6],[7],[8], and[9].
To overcome this, the spatial inductive bias must be restored. As explored by Fei Xie et al. in Mamba-Adaptor: State Space Model Adaptor for Visual Recognition, while auxiliary modules like multi-scale convolutions can inject local spatial awareness, the core sequence-processing engine itself requires a fundamentally multi-directional traversal strategy to capture global context.
Breaking the Causal Arrow: Bidirectional and Cross-Scanning
The simplest way to eliminate directional bias is to process the flattened sequence in both directions. Bidirectional scanning runs two parallel sequence models: one forward (from token $1$ to $N$) and one backward (from token $N$ to $1$).
Forward Path: [1] → [2] → [3] → [4] → [5]
Backward Path: [1] ← [2] ← [3] ← [4] ← [5]
Changqian Yu et al. utilize this approach in Scalable Diffusion Models with State Space Backbone to ensure that image patches in a diffusion generation pipeline can capture context from both ends of the flattened sequence. Similarly, Bohao Xing et al. apply bidirectional scanning to test-time training layers for facial action unit detection, mitigating the inherent bias of unidirectional sequence modeling.
Cross-Scanning (4-Way Traversal)
While bidirectional scanning solves the "past vs. future" problem along the horizontal raster line, it still fails to fix the vertical spatial discontinuity. To resolve this, architectures deploy cross-scanning or 4-way spatial scanning.
As detailed by Zeyu Wang et al. in Mamba YOLO: A Simple Baseline for Object Detection with State Space Model, the input feature map is unrolled along four distinct orthogonal paths:
- Horizontal Forward: Top-left to bottom-right (standard raster).
- Horizontal Backward: Bottom-right to top-left.
- Vertical Forward: Top-left to bottom-right via column-major ordering.
- Vertical Backward: Bottom-right to top-left via column-major ordering.
Horizontal Scan: Vertical Scan:
[1] → [2] → [3] [1] [4] [7]
↓ ↓ ↓
[4] → [5] → [6] [2] [5] [8]
↓ ↓ ↓
[7] → [8] → [9] [3] [6] [9]
By running four independent linear scans, every pixel is guaranteed to be immediately adjacent to its horizontal neighbors in the horizontal scans, and immediately adjacent to its vertical neighbors in the vertical scans. As Hashmat Shadab Malik et al. demonstrate, this 4-way coverage makes visual state space models remarkably robust to occlusions, because if an object is blocked along one scan path, its context can still be recovered via an orthogonal path.
Advanced Scanning Topologies
Beyond the standard 4-way cross-scan, researchers have engineered specialized scanning patterns tailored to specific data structures and computational constraints.
Diagonal and S-Shaped Scanning
The idea of altering the scan geometry dates back to early autoregressive models. In Pixel Recurrent Neural Networks, Aäron van den Oord et al. introduced diagonal scanning by skewing input feature maps, allowing a recurrent network to capture a complete, non-causal 2D receptive field. Modern architectures like the one proposed by Boyun Li et al. utilize "Nested S-shaped Scanning" to preserve local continuity and neighborhood boundaries during image restoration tasks.
Slice-and-Scan
To prevent the spatial misalignment that occurs when an entire image is linearized, Hanzhou Liu et al. propose a "slice-and-scan" strategy in XYScanNet: A State Space Model for Single Image Deblurring. Instead of flattening the entire 2D grid into a single 1D string, the model decomposes feature maps into discrete slices along the height and width dimensions, running specialized scanners within and across these slices to isolate local and global patterns cleanly.
Inter-Layer Flipping (Alternating Scans)
Executing four parallel scans within every single layer of a deep network is computationally expensive, multiplying the memory footprint and execution time of the linear backbone. To optimize this, models like Adventurer, designed by Feng Wang et al., shift the multi-directional logic from intra-layer to inter-layer.
Instead of scanning four directions simultaneously in one block, the network alternates the scanning direction between consecutive layers (e.g., Layer $L$ scans horizontally forward, Layer $L+1$ scans vertically forward).
Layer L (Horizontal): [Patch 1] → [Patch 2] → [Patch 3]
↓ (Feature Mixing)
Layer L+1 (Vertical): [Patch 1]
↓
[Patch 4]
Haowen Hou et al. leverage this inter-layer flipping in VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models, proving that information can effectively mix across the entire 2D space over the depth of the network. This hardware-friendly approach is also adopted by Benedikt Alkin et al. in Vision-LSTM: xLSTM as Generic Vision Backbone, alternating top-left-to-bottom-right sweeps with bottom-right-to-top-left sweeps across successive blocks.
Gating and Aggregation: Merging the Paths
Once an image has been processed through multiple directional paths, the resulting parallel feature representations must be fused back into a single unified 2D feature map. The choice of aggregation mechanism determines how effectively the model resolves conflicting directional contexts.
[Horizontal Features] ──┐
[Vertical Features] ──┼─→ [Adaptive Fusion Module] ─→ Integrated 2D Map
[Flipped Features] ──┘
- Direction-Wise Gating: In ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention, Bencheng Liao et al. implement an adaptive gating mechanism. The model computes a dynamic weight for each direction based on the local image content, allowing it to prioritize horizontal context when processing horizontal structures (like a horizon line) and vertical context for vertical structures (like a tree trunk).
- Channel-Wise Attention: Rather than treating each scan path equally, Boyun Li et al. use channel-wise attention to weight the aggregated paths. This allows the network to adaptively determine which directional perspectives are most relevant for restoring fine-grained details in specific regions of an image.
Generalization Beyond 2D Images
The principle of multi-directional scanning scales naturally to higher dimensions and irregular geometric domains where quadratic attention is prohibitively expensive.
3D Point Clouds and Grids
In 3D space, the lack of an inherent ordering is even more pronounced. Dingkang Liang et al. adapt the Mamba architecture to unstructured 3D data in PointMamba: A Simple State Space Model for Point Cloud Analysis by using multi-directional scanning combined with learnable indicators to distinguish between spatial perspectives. For continuous physical domains, such as solving partial differential equations (PDEs) on regular or irregular meshes, the Latent Mamba Operator employs four distinct paths across regular grids and bidirectional scans across irregular meshes to ensure isotropic information flow.
4D Light Fields
For complex multi-view data structures, the scanning paths can be mapped across entirely different physical dimensions. In LFTransMamba: A Hybrid Mamba-Transformer Model for Light Field Image Super-Resolution, Kai Jin et al. configure state space models to scan along multiple axes simultaneously: within individual spatial views, across different angular views, and at the pixel level.
By systematically mapping multi-directional paths to the underlying geometry of the data, multi-directional scanning successfully transforms 1D causal sequence processors into highly efficient, spatial-aware backbones capable of mastering non-causal visual worlds.