Overcoming the Spatial Sequential Mismatch in Visual Sequence Models

107 concepts93 papers1 proposal

Imagine trying to appreciate a painting by slicing it into thin horizontal strips, gluing them end-to-end into a single long ribbon, and reading it strictly from left to right.

If you encounter a pixel belonging to the bottom edge of a cloud, the pixel directly beneath it—the mountaintop—is now separated by the entire width of the canvas along your ribbon. Even worse, if your processing model is "causal," it can only look backward along the ribbon, leaving it completely blind to what lies ahead or directly below it in the real 2D space.

This exercise illustrates the spatial-sequential structural mismatch. Highly efficient sequence architectures—such as State Space Models (SSMs), LSTMs, and gated linear attention mechanisms—achieve linear computational complexity by processing data as a one-dimensional, unidirectional sequence. While this aligns perfectly with the temporal nature of text or audio, it fundamentally conflicts with the multi-directional, non-causal geometry of visual data.

The Geometry of the Ribbon: Why Flattening Fails

When we flatten a two-dimensional image grid into a one-dimensional sequence, we break spatial contiguity. As noted by Dongfei Yu et al. in Multi-level Attention Networks for Visual Question Answering, forcing a 2D image plane into a 1D sequence severely restricts a model's capacity to perform comprehensive spatial reasoning, frequently resulting in the loss of complex vertical and non-linear geometric dependencies.

This structural mismatch complicates the flow of information across multiple axes. Lucas Theis et al. recognized this early on in Generative Image Modeling Using Spatial LSTMs, demonstrating that standard 1D sequential architectures lack the specialized spatial inductive biases required to capture local correlations as effectively as multi-dimensional inputs.

The Causality Curse

The mismatch deepens when sequence models enforce a strict causal mask. In natural language, causality is a physical reality: a future word cannot influence a past word. In computer vision, however, spatial relationships are omnidirectional. A pixel's semantic meaning depends equally on its neighbors to the left, right, top, and bottom.

Hang Guo 0002 et al. formalize this challenge as the "causality curse" in MambaIRv2: Attentive State Space Restoration. They observe that unidirectional processing is fundamentally ill-suited for visual tasks like image restoration, where a token's state must be informed by surrounding context in all directions simultaneously. Forcing an autoregressive or causal constraint onto an image can severely degrade performance; for instance, Chunting Zhou et al. demonstrate in Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model that imposing a strict causal attention mask on image patches significantly impairs the quality of generated visuals.

Multi-Directional Scanning: Rebuilding 2D Context

To resolve this conflict without sacrificing the linear scaling laws that make sequence models attractive, researchers have redesigned how models traverse visual data. Instead of relying on a single raster scan, modern visual sequence models deploy multi-directional scanning strategies to reconstruct the 2D context.

Standard Raster Scan         Omni-Directional Scanning
┌───►───►───►───┐            ┌─►─┐ ┌─◄─┐ ┌───┐ ┌───┐
│               │            │   │ │   │ │ │ │ │ ▲ │
├───►───►───►───┤     vs     ▼   ▼ ▲   ▲ ▼ │ ▲ │ │ │
│               │            │   │ │   │ │ ▼ │ │ │ │
└───►───►───►───┘            └─►─┘ └─◄─┘ └───┘ └───┘
(Loses vertical context)     (Combines horizontal, vertical,
                              and diagonal passes)

For example, in VmambaIR: Visual State Space Model for Image Restoration, Yuan Shi et al. introduce an "Omni Selective Scan" mechanism. This approach processes the image across six distinct directions, allowing the model to aggregate long-range dependencies from multiple viewpoints simultaneously.

Similarly, Chenhongyi Yang et al. propose a continuous "zigzag" scanning process in PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition. Crucially, the authors pair this pattern with direction-aware updates, ensuring the model maintains explicit awareness of cardinal orientations during the state transition phase.

To break rigid directional biases entirely, Jie Xiao et al. introduce a stochastic scanning strategy in Non-causal selective state space model for image restoration. By randomly shuffling token sequences during training, the model bypasses the limitations of fixed directional updates, enabling pixels to utilize information from elements that would otherwise appear "later" in a deterministic sequence.

Content-Adaptive and Topological Traversals

While multi-directional geometric scans help bridge the spatial gap, they still treat the image as a rigid coordinate grid. A more advanced frontier involves matching the sequence traversal to the actual semantic content or topology of the image.

In Content-Aware Mamba for Learned Image Compression, Yunuo Chen et al. replace predefined scanning patterns with content-adaptive token reordering. By dynamically grouping semantically similar pixels together in the sequence and injecting global image prompts, they break Mamba's strict spatial causality, ensuring that related textures and objects update one another regardless of their physical distance on the grid.

Taking a mathematical approach to coordinate independence, Ali Bahri et al. develop a graph-based spectral ordering in Spectral State Space Model for Rotation-Invariant Visual Representation Learning. By mapping image patches to a graph and traversing them based on the eigenvectors of the graph Laplacian, the authors create a scanning order that naturally adapts to the image's underlying topology. This allows the model to handle multi-directional relationships without imposing an arbitrary coordinate system, yielding robust, rotation-invariant visual representations.

Correcting Coordinates with Multi-Dimensional Encodings

Even with multi-directional or adaptive scanning, a sequence model cannot fully comprehend 2D geometry if its coordinate system is fundamentally one-dimensional. As single author Bo Liu demonstrates in Transformer-based multiple instance learning network with 2D positional encoding for histopathology image classification, applying standard 1D positional encodings to flattened image data fails to represent the physical layout of complex spatial structures, such as biological tissue.

To preserve spatial geometry, researchers have re-engineered positional embeddings to natively support higher dimensions. Byeongho Heo et al. address this in Rotary Position Embedding for Vision Transformer by introducing a "RoPE-Mixed" framework. This method uses learnable frequencies to handle 2D spatial relationships, successfully capturing diagonal dependencies and preventing the localized attention artifacts common to simple axial decompositions.

1D Positional Encoding              2D/Lie Relative Encoding
[1]─[2]─[3]─[4]─[5]─[6]...          (X, Y) Coordinates preserved natively
                                    ┌───────┬───────┬───────┐
(Model assumes token 3              │ (1,1) │ (1,2) │ (1,3) │
 is far from token 6,               ├───────┼───────┼───────┤
 ignoring vertical adjacency)       │ (2,1) │ (2,2) │ (2,3) │
                                    └───────┴───────┴───────┘

Further expanding this concept, Sophie Ostmeier et al. introduce LieRE: Lie Rotational Positional Encodings. By leveraging Lie group theory, they generalize traditional rotary embeddings into high-dimensional, dense rotation matrices. This mathematical framework allows the model to learn continuous, multi-dimensional relationships between tokens, ensuring that whether a model processes data via attention or state-space equations, it retains an uncompromised awareness of the underlying physical space.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_spatial_mismatch_5460c876,
  title = {causal-spatial mismatch},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-spatial-mismatch/rdr_5460c876597d4e298cbd5afed49ca49f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_spatial_mismatch_5460c876_v1,
  title = {causal-spatial mismatch},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-spatial-mismatch/rdr_5460c876597d4e298cbd5afed49ca49f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake