In the world of Large Language Models (LLMs), the autoregressive (AR) paradigm is king. Models like GPT-4 generate text by predicting the "next token" in a strict, one-dimensional sequence. When researchers first applied this to images, they followed the same logic: flatten a 2D image into a 1D string of pixels or patches and predict them one-by-one, typically in a top-left to bottom-right "raster-scan" order.
However, as Keyu Tian and others have argued, this 1D approach is fundamentally at odds with the nature of visual data. Unlike a sentence, which has a clear temporal flow, an image is a 2D web of bidirectional dependencies. Forcing a model to finalize the top-left corner before it even "knows" what the bottom-right corner contains leads to a massive context lag and structural incoherence.
A new wave of research is moving beyond the raster scan, rethinking the order in which models "see" and "draw" the world.
The Raster-Scan Bottleneck
The primary critique of traditional AR models, such as those discussed in Deep Markov Random Fields for Image Modeling by Dahua Lin et al., is that they lack multi-directional spatial coherence. When a model predicts a pixel based only on its predecessors in a scanline, it ignores the "future" pixels that, in a 2D space, are actually its neighbors.
This "unidirectional constraint" creates several problems:
- Global Inconsistency: Models often commit to local details (like the texture of fur) before establishing the global structure (the shape of the animal), as noted in Generating High fidelity Images with subscale pixel Networks.
- Error Accumulation: A single mistake in an early token propagates through the entire sequence, a phenomenon Shuyang Gu et al. address in VQ-Diffusion.
- Inefficiency: Treating every patch with the same sequential priority forces the model to waste parameters on predictable backgrounds rather than focusing on semantically rich regions, a problem highlighted by Mengqi Huang in Masked Vector Quantization.
Coarse-to-Fine: Next-Scale Prediction
The most prominent alternative to "next-token" prediction is "next-scale" prediction. Instead of moving across the image spatially, the model moves through it hierarchically.
In Visual Autoregressive (VAR) Modeling, Keyu Tian introduces a paradigm where the model first generates a very low-resolution "thumbnail" (e.g., 1x1 or 4x4 tokens) and then uses that as context to predict the next, higher-resolution scale. This allows the model to establish a "global skeleton" before filling in the details.
This hierarchical approach is echoed in several other works:
- Hierarchical Pivots: Guangting Zheng and Yehao Li propose Hi-MAR, which uses low-resolution "pivot" tokens to guide the generation of dense details.
- Cluster-based Prediction: Ziyao Guo et al. use Coarse-to-Fine (CTF) Token Prediction to group similar tokens into clusters, predicting the "coarse" cluster identity before the fine-grained token.
- Dynamic Quantization: Mengqi Huang et al. in Towards Accurate Image Coding use a DQ-Transformer to generate images in a natural coarse-to-fine order, allocating more codes to complex regions.
Frequency and Spectral Ordering
Another way to achieve a "big picture first" generation is to move from the spatial domain to the frequency domain. Carlos Esteves at Google developed the Spectral Image Tokenizer (SIT), which uses Discrete Wavelet Transforms to order tokens by frequency.
Similarly, Zhihao Huang et al. introduced NFIG (Next-Frequency Image Generation). By generating low-frequency components first, these models ensure that the overall composition is sound before the high-frequency textures are added. This mimics human perception, where we often recognize the "gist" of a scene before its sharpest details.
Parallelism and Masked Modeling
If the problem is the strict 1D sequence, why not predict tokens in parallel? Huiwen Chang and Lu Jiang pioneered this with MaskGIT. Instead of a causal sequence, they use a bidirectional transformer that starts with a fully masked image and iteratively "unmasks" tokens based on the context of all other visible tokens.
This bidirectional approach effectively merges the benefits of AR and Diffusion. Sam Bond-Taylor and Peter Hessey take this further in Unleashing Transformers, using "discrete absorbing diffusion" to enable parallel token prediction. By treating generation as a denoising process of masked tokens, they bypass the slow, one-by-one nature of traditional AR while maintaining global context.
The Semantic Shift
Finally, some researchers argue that the order shouldn't be based on scale or frequency, but on meaning. Kaiyou Song et al. introduced SemAIM, which teaches models to focus on the main object first and then move toward the background. By ordering patches from high-to-low semantic importance, the model prioritizes the most informative regions of the image, leading to better visual representations.
Conclusion: A Unified Vision
The convergence of these ideas—multi-scale prediction, spectral ordering, and masked bidirectional modeling—suggests a future where the "sequence" in autoregressive modeling is no longer a spatial constraint, but a logical progression.
Whether it is Fluid using continuous tokens in a random order or AliTok aligning the tokenizer with a causal decoder, the goal is the same: to allow the model to "see" the whole canvas while it draws the first stroke. By breaking the raster scan, researchers are finally enabling AR models to scale in the visual domain with the same power and coherence they brought to language.