Visual Autoregressive Modeling via Next-Scale Prediction

This direction is merged into another direction: hierarchical image synthesis. Its last version stays readable and citable here.

Version 1 of 1100 papers121 conceptsfrozen 2026-10-04

Traditional autoregressive models, which power Large Language Models (LLMs), operate on a simple principle: predict the next token in a linear sequence. When applied to images, this usually means a "raster-scan" order—predicting pixels or patches from left-to-right, top-to-bottom. However, images are inherently two-dimensional and hierarchical, not linear. A pixel's meaning depends more on the global structure of the object it belongs to than on the pixel immediately to its left.

Visual Autoregressive (VAR) modeling, pioneered by Keyu Tian, Yi Jiang, and Bingyue Peng in Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, redefines this process. Instead of predicting the next token, the model predicts the next scale.

The Shift from Tokens to Scales

In a VAR framework, an image is not a flat string of tokens. Instead, it is represented as a multi-scale hierarchy—a pyramid of discrete token maps at increasing resolutions (e.g., $1\times1, 2\times2, 4\times4, \dots, R\times R$).

The autoregressive process follows this hierarchy:

  1. The model starts by predicting a very coarse $1\times1$ "thumbnail" that captures the global gist.
  2. Using that thumbnail as a prefix, it predicts the entire $2\times2$ token map in parallel.
  3. It continues this "next-scale prediction," using all previously generated lower-resolution maps to inform the next, higher-resolution layer.

This approach builds intuition by mimicking how a painter works: sketching the broad composition before filling in the fine-grained textures. As noted by Shenghao Xie and Wenqiang Zu in their survey on vision foundation models, this spectral ordering prevents "semantic drift," where a model loses track of the overall subject while focusing on local pixel transitions.

Why Next-Scale Prediction Works

The power of VAR lies in its efficiency and scaling behavior. By predicting entire resolution layers at once, the number of autoregressive steps drops from linear ($O(N)$ for $N$ patches) to logarithmic ($O(\log N)$ relative to the resolution). NestAR demonstrates that this nesting of scales allows for high-resolution synthesis in significantly fewer steps than traditional methods.

Furthermore, VAR models exhibit the same "power-law scaling" seen in LLMs. As you increase the model size and data, the performance improves predictably. This allows VAR to surpass Diffusion Transformers (DiT) in both image quality and speed. For instance, VARSR by Yunpeng Qu and colleagues achieves state-of-the-art image super-resolution while running up to 10x faster than diffusion-based alternatives.

Beyond Image Generation

The "next-scale" paradigm is proving to be a universal strategy for visual tasks:

  • Segmentation: Seg-VAR by rongkun Zheng treats segmentation as a hierarchical mask generation problem, resolving ambiguities in complex scenes by refining coarse global context into fine local details.
  • 3D Generation: SAR3D by Yongwei Chen applies this to 3D objects, enabling sub-second generation by predicting 3D token maps across scales.
  • Image Editing: AREdit and VAREdit leverage the scale hierarchy to preserve global structure (by keeping coarse tokens) while modifying specific details (by regenerating fine tokens).

Theoretical and Structural Refinements

The robustness of this idea is supported by theoretical work. Yifang Chen and Zhao Song provided the first rigorous proof in Universal Approximation of Visual Autoregressive Transformers that these models can approximate any complex image-to-image function with arbitrary precision.

Architecturally, researchers are finding that the hierarchical nature of VAR allows for unique optimizations. Anton Voronov and the Switti team discovered that because each scale already contains information from the previous one, standard causal masking is often redundant. Removing these constraints can reduce memory overhead and further accelerate inference. Similarly, HACK by Ziran Qin exploits the fact that different attention heads specialize in different scales, allowing for aggressive KV cache compression without losing image fidelity.

By aligning the autoregressive objective with the natural multi-scale structure of visual data, VAR models provide a faster, more scalable, and more intuitive path toward unified vision-language intelligence.

Papers in this version

Showing 60 of 100 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_visual_autoregressive_modeling_via_next__b7894722,
  title = {Visual Autoregressive Modeling via Next-Scale Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-autoregressive-modeling-via-next-scale-prediction/rdr_b78947223fc24790b81e6d898fb7ae39},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_visual_autoregressive_modeling_via_next__b7894722_v1,
  title = {Visual Autoregressive Modeling via Next-Scale Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-autoregressive-modeling-via-next-scale-prediction/rdr_b78947223fc24790b81e6d898fb7ae39/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}