Overcoming the Causality Curse in Visual Sequence Modeling

This direction is split into other directions: causal-spatial mismatch. Its last version stays readable and citable here.

Version 1 of 1196 papers220 conceptsfrozen 2026-10-04

Large Language Models (LLMs) have achieved remarkable success by treating text as a one-dimensional, unidirectional sequence. In this paradigm, a token only ever looks "backward" at what has already been written. However, as we move toward unified Vision-Language Models (VLMs), this "causal" constraint creates a fundamental mismatch. Unlike a sentence, an image has no natural beginning or end.

Researchers are now identifying this "causality curse"—the performance degradation that occurs when we force inherently spatial, non-causal visual data into the rigid sequential structures of autoregressive models and State Space Models (SSMs).

The Mismatch of Spatial and Sequential Logic

The core of the problem lies in how we "flatten" an image. To process an image with a Transformer or a Mamba block, we typically chop it into patches and arrange them in a line, often using a "raster scan" (left-to-right, top-to-bottom).

As Weihao Yu and Xinchao Wang argue in MambaOut: Do We Really Need Mamba for Vision?, this sequential order is arbitrary. In a causal model, a patch in the top-right corner cannot "see" a patch in the bottom-left, even though they might be part of the same object. This unidirectional processing prevents the model from capturing global context effectively. Luisa F. Polanía and colleagues point out in their work on GNNs that the order of processing should not affect the final interpretation of a scene, yet in causal models, it does.

Breaking the Mask with Future-Aware Attention

One way to fix this is to relax the "causal mask" that prevents tokens from looking ahead. In Rethinking Causal Mask Attention for Vision-Language Inference, Tao Huang and Chang Xu propose Future-Aware Attention.

Instead of forcing visual tokens to ignore everything to their "right," they allow tokens to peek at upcoming context during the prefill stage. By compressing this future information into "attention sinks," the model gains a holistic view of the image while maintaining the speed of autoregressive generation. Similarly, Chunting Zhou and the Transfusion team suggest that while text should remain causal, images should be treated with diffusion-like bidirectional attention to preserve their spatial integrity.

Multi-Directional Scanning in State Space Models

State Space Models like Mamba are prized for their linear complexity, but they are inherently causal—a state only depends on what came before it. To adapt this to vision, researchers have developed complex scanning strategies to "simulate" bidirectionality.

As detailed in A Survey on Visual Mamba by Dan Wang, models often scan an image in four or eight directions (up, down, left, right, and diagonals) to ensure every pixel eventually "sees" its neighbors. However, Jie Xiao and Xueyang Fu argue in their work on NCMamba that even these fixed scans are limiting. They propose stochastic scanning—randomly shuffling the sequence—to break the directional bias and capture non-local dependencies more effectively.

For image restoration, where local precision is vital, Hang Guo and Yulun Zhang introduced MambaIRv2. Their Attentive State-Space Equation (ASE) allows the model to perceive the entire image in a single scan, effectively giving Mamba the "global eyes" of a Transformer without the quadratic cost.

From Spatial Sequences to Spectral Causality

If spatial order is arbitrary, is there a natural sequence for images? Some researchers are looking toward the frequency domain.

In SpectralAR, Jiwen Lu and Wenzhao Zheng argue that while pixels have no order, frequencies do. They represent images as a 1D causal sequence of frequency components, moving from coarse global structures to fine details. This "spectral causality" aligns better with how images are composed, allowing autoregressive models to generate high-quality visuals with far fewer tokens.

Similarly, DCTMamba by Xi Wang uses the Discrete Cosine Transform to establish a "coarse-to-fine" scanning order, providing the causality Mamba needs while respecting the multi-scale nature of visual information.

The Path Forward: Native Multimodality

The recurring theme across these 151 papers is that the "causality" we use for language is a borrowed tool, not a native one for vision. Whether through parallel row decoding in ZipAR or the use of Spectral State Space Models for rotation invariance, the field is moving toward architectures that treat space and time as distinct but integrated dimensions. By breaking the causality curse, we move closer to models that can truly "see" an image all at once, just as humans do.

Papers in this version

Showing 60 of 196 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_overcoming_the_causality_curse_in_visual_528bbe5f,
  title = {Overcoming the Causality Curse in Visual Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-causality-curse-in-visual-sequence-modeling/rdr_528bbe5f3ade4c92b12568766c22aafb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_overcoming_the_causality_curse_in_visual_528bbe5f_v1,
  title = {Overcoming the Causality Curse in Visual Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-causality-curse-in-visual-sequence-modeling/rdr_528bbe5f3ade4c92b12568766c22aafb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}