Autoregressive (AR) models like GPT have dominated natural language processing, but bringing that same "next-token prediction" magic to images has been challenging. While recent breakthroughs like the Visual Autoregressive (VAR) model moved away from simple pixel-by-pixel generation toward "next-scale" prediction, they still hit a computational ceiling due to the quadratic complexity of standard self-attention.
In their new paper, researchers from Johns Hopkins, UC Berkeley, Google, and UC Santa Cruz introduce M-VAR, a framework that optimizes this process by decoupling how a model looks at local details versus global structure.
The Efficiency-Importance Mismatch
Traditional autoregressive models for images usually flatten a 2D image into a 1D string of tokens, ignoring the inherent 2D structure of visual data. The VAR model solved this by generating images scale-by-scale. However, VAR uses standard global self-attention. As the model moves to finer scales, the sequence length grows, and the computational cost explodes.
The authors identified an "efficiency-importance" mismatch: interactions within a scale (intra-scale) are vital for image quality but involve short sequences, while interactions between scales (inter-scale) involve massive sequences but are less complex to model. Their analysis showed that inter-scale interactions consume up to 76% of the compute despite contributing less to the final attention distribution.
The M-VAR Framework
The core innovation of M-VAR is the decoupling of these two modeling tasks:
- Intra-scale Modeling (The "Detail" Specialist): To capture local spatial dependencies and fine-grained textures within a single resolution, the authors retain bidirectional self-attention. Since each scale is processed independently here, the sequences are short, making the quadratic cost of attention manageable.
- Inter-scale Modeling (The "Global" Architect): To maintain consistency across different resolutions, the authors use Mamba. Mamba is a State Space Model (SSM) with linear complexity, meaning it can handle the very long sequences created by concatenating all image scales without the massive memory overhead of traditional Transformers.
By combining these, M-VAR gets the high-fidelity benefits of attention where it matters most and the linear-speed benefits of Mamba for the heavy lifting.
Key Results and Performance
M-VAR sets a new bar for autoregressive image generation on the industry-standard ImageNet benchmark:
- Superior Quality: M-VAR-d32 achieved a Fréchet Inception Distance (FID) of 1.78 on ImageNet 256×256. This beats the popular Diffusion Transformer (DiT-XL/2) by 0.49 and the original VAR-d30 by 0.19.
- Better Scaling: An M-VAR model with 1.5 billion parameters outperformed a 2.0 billion parameter VAR model while being 1.2x faster during inference.
- Inference Efficiency: As shown in the performance plots, M-VAR consistently achieves lower (better) FID scores at significantly faster inference speeds than both diffusion-based and previous autoregressive models.
Conclusion
While M-VAR significantly reduces the computational burden of inter-scale modeling, it still relies on a pre-trained VQ-VAE to tokenize images, meaning the quality is ultimately capped by the tokenizer's reconstruction ability. However, by decoupling scale-wise modeling, M-VAR provides a more efficient and high-quality path for autoregressive image generation, surpassing both previous AR models and popular diffusion-based approaches like DiT.
Summary updated