M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation

Sucheng Ren, Yaodong Yu, Nataniel Ruiz, Feng Wang, Alan L. Yuille, Cihang Xie

Autoregressive (AR) models like GPT have dominated natural language processing, but bringing that same "next-token prediction" magic to images has been challenging. While recent breakthroughs like the Visual Autoregressive (VAR) model moved away from simple pixel-by-pixel generation toward "next-scale" prediction, they still hit a computational ceiling due to the quadratic complexity of standard self-attention.

In their new paper, researchers from Johns Hopkins, UC Berkeley, Google, and UC Santa Cruz introduce M-VAR, a framework that optimizes this process by decoupling how a model looks at local details versus global structure.

The Efficiency-Importance Mismatch

Traditional autoregressive models for images usually flatten a 2D image into a 1D string of tokens, ignoring the inherent 2D structure of visual data. The VAR model solved this by generating images scale-by-scale. However, VAR uses standard global self-attention. As the model moves to finer scales, the sequence length grows, and the computational cost explodes.

The authors identified an "efficiency-importance" mismatch: interactions within a scale (intra-scale) are vital for image quality but involve short sequences, while interactions between scales (inter-scale) involve massive sequences but are less complex to model. Their analysis showed that inter-scale interactions consume up to 76% of the compute despite contributing less to the final attention distribution.

The M-VAR Framework

The core innovation of M-VAR is the decoupling of these two modeling tasks:

  1. Intra-scale Modeling (The "Detail" Specialist): To capture local spatial dependencies and fine-grained textures within a single resolution, the authors retain bidirectional self-attention. Since each scale is processed independently here, the sequences are short, making the quadratic cost of attention manageable.
  2. Inter-scale Modeling (The "Global" Architect): To maintain consistency across different resolutions, the authors use Mamba. Mamba is a State Space Model (SSM) with linear complexity, meaning it can handle the very long sequences created by concatenating all image scales without the massive memory overhead of traditional Transformers.
The M-VAR architecture decouples intra-scale attention from inter-scale Mamba modeling to optimize for both quality and speed.

By combining these, M-VAR gets the high-fidelity benefits of attention where it matters most and the linear-speed benefits of Mamba for the heavy lifting.

Key Results and Performance

M-VAR sets a new bar for autoregressive image generation on the industry-standard ImageNet benchmark:

  • Superior Quality: M-VAR-d32 achieved a Fréchet Inception Distance (FID) of 1.78 on ImageNet 256×256. This beats the popular Diffusion Transformer (DiT-XL/2) by 0.49 and the original VAR-d30 by 0.19.
  • Better Scaling: An M-VAR model with 1.5 billion parameters outperformed a 2.0 billion parameter VAR model while being 1.2x faster during inference.
  • Inference Efficiency: As shown in the performance plots, M-VAR consistently achieves lower (better) FID scores at significantly faster inference speeds than both diffusion-based and previous autoregressive models.
M-VAR achieves a superior FID-speed trade-off compared to VAR and Diffusion models (DiT), reaching lower error in less time.

Conclusion

While M-VAR significantly reduces the computational burden of inter-scale modeling, it still relies on a pre-trained VQ-VAE to tokenize images, meaning the quality is ultimately capped by the tokenizer's reconstruction ability. However, by decoupling scale-wise modeling, M-VAR provides a more efficient and high-quality path for autoregressive image generation, surpassing both previous AR models and popular diffusion-based approaches like DiT.

Summary updated

Abstract

There exists recent work in computer vision, named VAR, that proposes a new autoregressive paradigm for image generation. Diverging from the vanilla next-token prediction, VAR structurally reformulates the image generation into a coarse to fine next-scale prediction. In this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into \textit{intra-scale modeling}, which captures local spatial dependencies within each scale, and \textit{inter-scale modeling}, which models cross-scale relationships progressively from coarse-to-fine scales. This decoupling structure allows to rebuild VAR in a more computationally efficient manner. Specifically, for intra-scale modeling -- crucial for generating high-fidelity images -- we retain the original bidirectional self-attention design to ensure comprehensive modeling; for inter-scale modeling, which semantically connects different scales but is computationally intensive, we apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. We term this new framework M-VAR. Extensive experiments demonstrate that our method outperforms existing models in both image quality and generation speed. For example, our 1.5B model, with fewer parameters and faster inference speed, outperforms the largest VAR-d30-2B. Moreover, our largest model M-VAR-d32 impressively registers 1.78 FID on ImageNet 256$\times$256 and outperforms the prior-art autoregressive models LlamaGen/VAR by 0.4/0.19 and popular diffusion models LDM/DiT by 1.82/0.49, respectively. Code is avaiable at \url{https://github.com/OliverRensu/MVAR}.
BibTeX
@misc{ren2024m,
  author = {Sucheng Ren and Yaodong Yu and Nataniel Ruiz and Feng Wang and Alan L. Yuille and Cihang Xie},
  title = {{M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation}},
  year = {2024},
  eprint = {2411.10433},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.10433},
}
Figure