Autoregressive (AR) models, the backbone of modern large language models, have recently been adapted for visual synthesis. By quantizing images into discrete tokens, models like CogView treat image generation as a sequence prediction task. However, this approach faces a fundamental "sequential generation bottleneck." While text is inherently one-dimensional, images are high-dimensional grids. Predicting these grids token-by-token—typically in a top-left to bottom-right "raster-scan" order—results in extreme inference latency that scales linearly with the number of pixels.
The Raster-Scan Bottleneck
In traditional autoregressive frameworks, each token must wait for the full forward pass of all preceding tokens. As Dahl et al. and Bond-Taylor et al. observe, this sequential dependency prevents parallelization during inference. For high-resolution images, where the number of tokens can reach into the thousands, the computational cost becomes prohibitive. Tian notes that this complexity can reach $O(n^6)$ for high-resolution content, making real-time applications nearly impossible for standard AR architectures.
The research community has responded by developing mechanisms to break this linear dependency, moving from rigid sequential decoding to parallelized and multi-scale strategies.
From Tokens to Scales
One of the most significant shifts in intuition is moving away from predicting individual tokens to predicting entire "scales" of an image. Tian proposed Visual Autoregressive (VAR) modeling, which replaces the raster-scan order with a coarse-to-fine "next-scale" prediction. Instead of predicting the next pixel, the model predicts the next resolution level of the image. This allows the model to capture global structure first and then fill in details, achieving 20x-45x faster inference than traditional methods.
Similarly, Zhang et al. introduced NextFlow, which uses a hierarchical approach to generate 1024×1024 images in seconds. By treating the image as a hierarchy rather than a flat sequence, these models leverage the spatial redundancy of visual data, where global context informs local details simultaneously.
Parallelizing the Sequence
Another strategy involves modifying the decoding process to allow multiple tokens to be predicted at once.
- Iterative Parallel Decoding: Chang et al. (MaskGIT) introduced a paradigm where the model predicts all tokens in parallel, starting from a fully masked grid and iteratively refining the most confident predictions. This bidirectional approach achieves up to a 64x speedup over sequential models.
- Randomized and Locality-Aware Decoding: Li et al. developed ARPG, which decouples positional guidance from content to enable randomized parallel decoding. Further, Zhang et al. proposed Locality-aware Parallel Decoding (LPD), which uses a generation schedule based on spatial proximity, reducing sampling steps from hundreds to as few as twenty.
- Speculative Decoding: Borrowing from language modeling, Teng introduced Speculative Jacobi Decoding (SJD). This training-free method allows a model to decode multiple image tokens in parallel by checking for probabilistic convergence, effectively doubling the speed of existing text-to-image AR models.
Hybrid Representations and Continuous Tokens
The bottleneck is also addressed by rethinking the tokens themselves. While discrete tokens (like those in VQ-VAE) simplify modeling, they often lead to long sequences. Liu et al. (SAR) proposed predicting "sets" of tokens to retain the benefits of KV caching while accelerating generation by 60x.
More recently, researchers have explored continuous tokens to bridge the gap between AR and diffusion models. Han et al. (NextStep-1) and Yu et al. (VideoMAR) use continuous representations combined with flow-matching or diffusion losses. This allows the model to maintain the logical reasoning of a transformer while benefiting from the parallel "denoising" capabilities of diffusion, significantly reducing the number of steps required for high-fidelity video and image synthesis.
Efficiency in Specialized Tasks
The push for efficiency extends beyond general synthesis to specific vision tasks. Wang et al. (ARGenSeg) demonstrated that treating image segmentation as an AR task can achieve a 4x speedup by using multi-scale parallel token prediction. In the realm of object discovery, the authors of MOGENT achieved a 67x speedup over traditional AR models by replacing sequential decoders with parallel masked transformers.
By moving away from the strict "one-token-at-a-time" philosophy, these diverse approaches collectively transform the autoregressive transformer from a slow, sequential processor into a high-speed, parallelized engine for visual generation.