Scalable Pre-training of Large Autoregressive Image Models

Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, Armand Joulin

TL;DR

AIM (Autoregressive Image Models) demonstrates that the "LLM recipe"—scaling large Transformer models with a simple next-token prediction objective—works remarkably well for computer vision. By pre-training Vision Transformers up to 7 billion parameters on 2 billion uncurated images, the authors show that visual features exhibit the same predictable scaling laws as text, achieving state-of-the-art performance without the complex stability tricks required by previous methods.

What Problem Does This Paper Address?

In Natural Language Processing, Large Language Models (LLMs) have revolutionized the field by showing that performance scales consistently with model capacity and data size using a simple autoregressive objective: predicting the next word.

Computer vision has struggled to find a similarly "clean" scaling path. Current state-of-the-art vision models typically rely on contrastive learning (like CLIP or DINO), which requires complex data curation and training "tricks" (e.g., LayerScale, stochastic depth) to remain stable at scale, or masked image modeling (like MAE), which hasn't always shown the same clear scaling laws as LLMs. The authors ask a fundamental question: Can we simply treat image patches like words and use the autoregressive objective to build scalable, general-purpose vision models?

Methods

The authors introduce AIM, a collection of vision models ranging from 600M to 7B parameters. The core approach is surprisingly simple:

  1. Architecture: They use a standard Vision Transformer (ViT) backbone.
  2. Objective: The model is trained to predict the next patch in an image given the preceding ones, following a raster (row-major) order.
  3. Prefix Causal Attention: Standard LLMs use a causal mask so tokens only look backward. However, downstream vision tasks benefit from bidirectional context. AIM uses a "prefix" during pre-training—a set of initial patches that can all see each other—allowing the model to adapt more easily to bidirectional attention during downstream tasks.
  4. MLP Prediction Head: Instead of predicting pixels directly from the ViT trunk, they use a heavy MLP head. This prevents the trunk features from becoming too specialized in low-level pixel reconstruction, keeping them "semantic" for downstream use.
  5. Loss Function: They use a normalized pixel-level regression loss (MSE), finding it superior to discrete tokens (like those used in VQ-VAE).
AIM pre-training overview. Images are split into patches, processed by a causal Transformer, and an MLP predicts the next patch.

Datasets

  • Pre-training: The primary dataset is DFN-2B, a subset of 2 billion uncurated images from the DataComp collection. They also use a mixture called DFN-2B+, which includes a small percentage of ImageNet-1k data to improve downstream alignment.
  • Evaluation: The models are tested across a diverse suite of 15 benchmarks, including standard ones like ImageNet-1k, Cifar10/100, and specialized datasets like iWildCam (natural environments) and Camelyon17 (medical imaging).

Key Insights and Ideas

  • The Scaling Law Holds: Just like in LLMs, AIM's downstream performance is a direct function of the pre-training validation loss. As you increase parameters and data, the loss drops, and accuracy on benchmarks rises predictably.
  • No Saturation: Even at 7 billion parameters and 2 billion images, the authors observe no signs of performance plateauing. This suggests that even larger vision models are viable.
  • Simplicity is Stability: Unlike many self-supervised vision methods, AIM does not require specialized optimization techniques like LayerScale or QK-Norm to prevent training divergence at scale.
  • Feature Location: Interestingly, the "best" features for downstream tasks aren't found in the very last layer of the model (which is busy predicting pixels) but roughly two-thirds of the way through the trunk.
Figure 1. AIM scaling behavior (Left) As we scale the capacity of AIM, we observe improved performance for the pre-training objective which directly correlates with stronger downstream performance. (Right) AIM exhibits stronger downstream performance when trained using larger sets of uncurated web data [32, 33]. The downstream performance is the average attentive probe top-1 accuracy over a diverse set of 15 image recognition benchmarks. All models are trained for the same number of updates.

Key Results

  • ImageNet Performance: AIM-7B achieves 84.0% top-1 accuracy on ImageNet-1k using a frozen trunk (Attentive Probe).
  • Generative Superiority: AIM significantly outperforms other generative pre-training methods like MAE and BEiT at similar scales.
  • Competitive with Joint-Embedding: While slightly behind DINOv2 (which uses higher resolution and more complex training), AIM narrows the gap significantly while using a much simpler, more scalable objective.
  • Adaptability: Using Low-Rank Adaptation (LoRA), AIM-7B reaches 86.3% on ImageNet-1k by fine-tuning only 0.1% of its parameters.
Downstream performance across layers. The highest quality features typically reside in shallower layers than the last.

Limitations and Future Work

  • Sample Efficiency: While AIM scales beautifully, it is less sample-efficient than Masked Autoencoders (MAE). It requires massive amounts of data to reach its full potential.
  • Resolution: The current study focuses on fixed-resolution patches. Exploring how autoregressive models handle multi-scale or high-resolution inputs is a natural next step.
  • Traversal Patterns: While raster order works best currently, the authors note that the "difficulty" of prediction varies across the image. Future work could explore more sophisticated ways to order or group patches for prediction.
  • Beyond Generative: The authors suggest that the synergy between autoregressive objectives and Transformers might eventually lead to a unified architecture for both vision and language.

Summary updated

Abstract

This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling properties. Specifically, we highlight two key findings: (1) the performance of the visual features scale with both the model capacity and the quantity of data, (2) the value of the objective function correlates with the performance of the model on downstream tasks. We illustrate the practical implication of these findings by pre-training a 7 billion parameter AIM on 2 billion images, that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at this scale, we observe no sign of saturation in performance, suggesting that AIM potentially represents a new frontier for training large-scale vision models. The pre-training of AIM is similar to the pre-training of LLMs, and does not require any image-specific strategy to stabilize the training at scale.
BibTeX
@inproceedings{elnouby2024scalable,
  author = {Alaaeldin El-Nouby and Michal Klein and Shuangfei Zhai and Miguel Angel Bautista and Alexander Toshev and Vaishaal Shankar and Joshua M Susskind and Armand Joulin},
  title = {{Scalable Pre-training of Large Autoregressive Image Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=c92KDfEZTg},
}
Figure