Autoregressive models have achieved dominant success in natural language processing by predicting the "next token" in a sequence. When researchers extended this paradigm to computer vision, they naturally treated images like text: flattening two-dimensional grids of pixels or latent codes into a one-dimensional sequence, typically processed in a rigid, top-left to bottom-right "raster-scan" order.
However, as Zhicong Tang et al. note in BAR: Refactor the Basis of Autoregressive Visual Generation, images are not books. Visual data possesses multi-directional, non-linear spatial dependencies that a strict linear sequence cannot naturally capture. Forcing a one-dimensional causal chain onto multidimensional data introduces an artificial directional bias—often called the raster-scan constraint—that fundamentally limits how models reason about space.
The Cost of Tunnel Vision
To understand why the raster-scan constraint hurts visual synthesis, consider how a standard causal transformer generates an image. Because it uses a causal attention mask, a token at any given position can only attend to tokens that preceded it in the serialization order.
This creates a form of structural "tunnel vision." As Patrick Esser et al. explain in ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis, a model generating the top half of an image is entirely blind to the future spatial context of the bottom half. It is forced to commit to fine, local pixel details before the global layout or downstream structures are even established, frequently leading to global structural inconsistencies.
Rigid Raster Scan (Unidirectional Context):
[Token 1] → [Token 2] → [Token 3] → [Token 4] ...
(Early tokens cannot see downstream spatial context)
Stochastic / Bidirectional Generation (Global Context):
[Token 1] ↔ [Token 2] ↔ [Token 3] ↔ [Token 4]
(Tokens dynamically condition on surrounding spatial context)
This unidirectional constraint severely limits the model's flexibility in practical applications:
- Downstream Editing Inflexibility: If a user wants to perform outpainting or fill in an arbitrary spatial gap, a fixed scanline model struggles. As Chris Rockwell et al. observe in PixelSynth: Generating a 3D-Consistent Experience from a Single Image, a fixed top-to-bottom sequence cannot effectively utilize context from pixels that reside at the outer edges or "downstream" of the generation path.
- Spurious Local Biases: Rongzhen Zhao argues in Slot Attention with Re-Initialization and Self-Distillation that sequential decoders tend to over-rely on local scan-line correlations rather than learning true, robust object representations.
- Layout Failures: In structured domains like architectural design, placing initial components without accounting for the spatial requirements of future elements causes layouts to violate complex connectivity constraints (Jiachen Liu et al. in End-to-End Graph-Constrained Vectorized Floorplan Generation with Panoptic Refinement).
To overcome these limitations, a growing body of research focuses on breaking the rigid 1D sequence, allowing models to leverage bidirectional and global context.
Breaking the Sequence via Stochastic Permutations
The most direct way to eliminate raster-scan bias is to train the model on randomized token sequences rather than a fixed path. If the generation order is completely unpredictable during training, the model can no longer rely on a fixed directional history. Instead, it must learn to predict the value of any arbitrary pixel given any surrounding spatial context.
Ziqi Pang et al. implement this intuition in RandAR: Decoder-only Autoregressive Visual Generation in Random Orders by prepending a position instruction token to each visual token. Because the sequence order is randomized, the model naturally learns multi-directional dependencies, unlocking zero-shot inpainting and outpainting without requiring specialized architectures.
Similarly, Qi Yu et al. in Exploring Stochastic Autoregressive Image Modeling for Visual Representation and Lijie Fan et al. in Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens demonstrate that stochastic ordering allows large-scale autoregressive models to capture bidirectional context effectively, leading to significantly better spatial reasoning and object placement.
This concept extends beyond generation to image restoration. In Non-causal selective state space model for image restoration, Jie Xiao et al. introduce a stochastic scanning strategy to linear-complexity state space models (like Mamba). Shuffling the scanning trajectories prevents the model from enforcing an artificial spatial hierarchy, allowing it to exploit contextually related but spatially distant pixels for denoising and deblurring.
Iterative Masking and Discrete Diffusion
An alternative to shuffling the sequence is to abandon the one-pass, token-by-token causal chain entirely. Instead, generation can be framed as an iterative "denoising" or "mask-and-replace" task over a fully bidirectional transformer.
Huiwen Chang et al. pioneered this shift with MaskGIT: Masked Generative Image Transformer. Rather than predicting a single token per step from left to right, MaskGIT starts with a completely masked canvas and predicts all tokens in parallel. At each step, the model keeps the most confident predictions and re-masks the rest. Because the attention mechanism is fully bidirectional, the model uses the evolving global layout to refine local details concurrently.
Iterative Parallel Decoding (e.g., MaskGIT):
Step 1 (Global Layout): [ M ] [ M ] [ M ] [ M ] → Predicts rough structure
Step 2 (Refinement): [ A ] [ M ] [ M ] [ B ] → Fills in confident tokens
Step 3 (Final Details): [ A ] [ C ] [ D ] [ B ] → Completes the image
This iterative parallel decoding closely connects to discrete diffusion models. As Shuyang Gu et al. show in Vector Quantized Diffusion Model for Text-to-Image Synthesis, treating generation as a bidirectional denoising process over discrete vector-quantized tokens eliminates the error accumulation inherent to long autoregressive chains.
By replacing sequential constraints with discrete diffusion priors, frameworks like those proposed by Minghui Hu et al. in Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation and Sam Bond-Taylor et al. in Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes achieve superior global consistency while accelerating sampling speeds up to 100× compared to traditional pixel-space or causal autoregressive models.
Coarse-to-Fine Hierarchies
If a model must generate data sequentially, it can still avoid the pitfalls of the raster scan by changing what the sequence represents. Instead of scanning across a spatial grid, the sequence can progress through a structural hierarchy—moving from a low-resolution global abstraction to high-resolution local details.
The authors of Generating High fidelity Images with subscale pixel Networks and Multidimensional Upscaling addressed this by slicing images into interleaved "subscales." By generating a low-resolution version of the entire canvas first, the model establishes a holistic structural anchor before conditioning subsequent steps on this global context.
This "draft-then-detail" intuition is formalized by the authors of Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer, who decouple the generation process into two distinct phases: a bidirectional transformer first sketches a diverse, coarse global "draft," and a subsequent network revises and fills in the fine textures.
By restructuring the generation sequence around scale and abstraction rather than arbitrary spatial coordinates, these hierarchical approaches ensure that the model never has to make fine-grained local commitments in a structural vacuum.
Key Foundational Work
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021) — contributed 1 concept elements to this direction.
PixelTransformer: Sample Conditioned Signal Generation (2021) — contributed 1 concept elements to this direction.
Generating High fidelity Images with subscale pixel Networks and Multidimensional Upscaling (2019) — contributed 1 concept elements to this direction.