Scaling Visual Generation with Autoregressive Transformers

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2239 papers243 conceptsfrozen 2026-10-04

For years, the recipe for high-fidelity image generation was dominated by two ingredients: the U-Net architecture and the diffusion process. While successful, this paradigm struggled to inherit the predictable scaling laws and unified modeling power that Large Language Models (LLMs) brought to text. A new direction has emerged that swaps convolutional backbones for Transformers and recasts image generation as a sequence modeling task, effectively treating pixels—or their latent representations—as a language to be predicted.

The Shift from U-Nets to Transformers

The transition began with the realization that the inductive biases of convolutions, while helpful for small models, might limit the scalability of generative models. William Peebles and Saining Xie demonstrated in Scalable Diffusion Models with Transformers (DiT) that replacing the U-Net with a Transformer backbone allows image generators to follow clear scaling laws. By increasing model depth or decreasing the size of image "patches," performance improves predictably, mirroring the progress seen in LLMs.

However, simply swapping the backbone was only the first step. Researchers like Shanghua Gao and Pan Zhou found that these Transformers could be made even more efficient through masked modeling. Their work on MDTv2 introduces a masked latent modeling scheme that forces the model to learn contextual relations between different image parts, significantly accelerating training speed.

Beyond Raster Scans: Next-Scale Prediction

If images are to be treated like text, how should we "read" them? The traditional "raster-scan" order (top-left to bottom-right) is natural for text but arbitrary for 2D images. Keyu Tian and colleagues proposed a fundamental shift in Visual Autoregressive Modeling (VAR). Instead of predicting the next pixel, the model predicts the "next scale."

The intuition is that images are inherently hierarchical. By generating a coarse, low-resolution version of an image first and then autoregressively predicting higher-resolution details, models can capture global structure before committing to fine textures. This "next-scale" approach allows GPT-style models to surpass Diffusion Transformers in both speed and quality, offering a path toward unified multimodal modeling.

Tokenization as the Key

The success of these autoregressive models hinges on how images are converted into discrete "tokens." If the vocabulary is poor, the "language" of the image becomes unintelligible. Lijun Yu and Vighnesh Birodkar argued in MAGVIT-v2 that the bottleneck in visual LLMs wasn't the Transformer itself, but the tokenizer. By introducing lookup-free quantization, they enabled LLMs to outperform diffusion models in fidelity.

Further refinements to this "vocabulary" include the Discriminative Codebook Prior Extractor (DCPE) by Longxiang Tang, which uses density-aware clustering to better organize image tokens, and IAR2 by Teng Hu, which decouples image representations into separate semantic and detail codebooks.

Efficiency and the Removal of Guidance

A major hurdle for both diffusion and autoregressive models is the reliance on Classifier-Free Guidance (CFG) at inference time, which requires running the model twice for every step. To solve this, Huayu Chen and Jun Zhu developed Condition Contrastive Alignment (CCA). By fine-tuning the model to align its conditional and unconditional distributions, they achieved CFG-level quality without the 2x inference cost. Similarly, Zhicong Tang and Jianmin Bao introduced Model-guidance, which bakes the guidance logic directly into the training objective.

The Frontier: State Space Models and Open-Domain Synthesis

While Transformers are the current standard, the quadratic cost of attention remains a challenge for high-resolution generation. Researchers are exploring alternatives like State Space Models (SSMs). Jiatao Gu and Alexander M Rush introduced DIFFUSSM, replacing attention with gated Bidirectional SSMs to reduce FLOPs. Similarly, Xin Li proposed AiM, using the Mamba architecture for autoregressive generation.

The ultimate goal of this research direction is to move beyond class-conditional generation (e.g., "generate a cat") to full, open-domain text-to-image synthesis. As noted in Lumina-mGPT by Dongyang Liu, unifying visual generation with the semantic depth of LLMs requires not just architectural changes, but massive-scale alignment between natural language and visual token sequences. The field is currently bridging this gap, proving that the same "next-token" logic that powers human-like conversation can also paint human-like art.

Papers in this version

Showing 60 of 239 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 134 papers
  2. Version 1 · 2026-10-04 · 239 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_text_to_image_scalability_ddb6b4af,
  title = {text-to-image scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/text-to-image-scalability/rdr_ddb6b4af809c48abb543a964664bb970},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_visual_generation_with_autoregre_ddb6b4af_v1,
  title = {Scaling Visual Generation with Autoregressive Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-visual-generation-with-autoregressive-transformers/rdr_ddb6b4af809c48abb543a964664bb970/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}