TL;DR
AIM (Autoregressive Image Models) demonstrates that the "LLM recipe"—scaling large Transformer models with a simple next-token prediction objective—works remarkably well for computer vision. By pre-training Vision Transformers up to 7 billion parameters on 2 billion uncurated images, the authors show that visual features exhibit the same predictable scaling laws as text, achieving state-of-the-art performance without the complex stability tricks required by previous methods.
What Problem Does This Paper Address?
In Natural Language Processing, Large Language Models (LLMs) have revolutionized the field by showing that performance scales consistently with model capacity and data size using a simple autoregressive objective: predicting the next word.
Computer vision has struggled to find a similarly "clean" scaling path. Current state-of-the-art vision models typically rely on contrastive learning (like CLIP or DINO), which requires complex data curation and training "tricks" (e.g., LayerScale, stochastic depth) to remain stable at scale, or masked image modeling (like MAE), which hasn't always shown the same clear scaling laws as LLMs. The authors ask a fundamental question: Can we simply treat image patches like words and use the autoregressive objective to build scalable, general-purpose vision models?
Methods
The authors introduce AIM, a collection of vision models ranging from 600M to 7B parameters. The core approach is surprisingly simple:
- Architecture: They use a standard Vision Transformer (ViT) backbone.
- Objective: The model is trained to predict the next patch in an image given the preceding ones, following a raster (row-major) order.
- Prefix Causal Attention: Standard LLMs use a causal mask so tokens only look backward. However, downstream vision tasks benefit from bidirectional context. AIM uses a "prefix" during pre-training—a set of initial patches that can all see each other—allowing the model to adapt more easily to bidirectional attention during downstream tasks.
- MLP Prediction Head: Instead of predicting pixels directly from the ViT trunk, they use a heavy MLP head. This prevents the trunk features from becoming too specialized in low-level pixel reconstruction, keeping them "semantic" for downstream use.
- Loss Function: They use a normalized pixel-level regression loss (MSE), finding it superior to discrete tokens (like those used in VQ-VAE).
Datasets
- Pre-training: The primary dataset is DFN-2B, a subset of 2 billion uncurated images from the DataComp collection. They also use a mixture called DFN-2B+, which includes a small percentage of ImageNet-1k data to improve downstream alignment.
- Evaluation: The models are tested across a diverse suite of 15 benchmarks, including standard ones like ImageNet-1k, Cifar10/100, and specialized datasets like iWildCam (natural environments) and Camelyon17 (medical imaging).
Key Insights and Ideas
- The Scaling Law Holds: Just like in LLMs, AIM's downstream performance is a direct function of the pre-training validation loss. As you increase parameters and data, the loss drops, and accuracy on benchmarks rises predictably.
- No Saturation: Even at 7 billion parameters and 2 billion images, the authors observe no signs of performance plateauing. This suggests that even larger vision models are viable.
- Simplicity is Stability: Unlike many self-supervised vision methods, AIM does not require specialized optimization techniques like LayerScale or QK-Norm to prevent training divergence at scale.
- Feature Location: Interestingly, the "best" features for downstream tasks aren't found in the very last layer of the model (which is busy predicting pixels) but roughly two-thirds of the way through the trunk.
Key Results
- ImageNet Performance: AIM-7B achieves 84.0% top-1 accuracy on ImageNet-1k using a frozen trunk (Attentive Probe).
- Generative Superiority: AIM significantly outperforms other generative pre-training methods like MAE and BEiT at similar scales.
- Competitive with Joint-Embedding: While slightly behind DINOv2 (which uses higher resolution and more complex training), AIM narrows the gap significantly while using a much simpler, more scalable objective.
- Adaptability: Using Low-Rank Adaptation (LoRA), AIM-7B reaches 86.3% on ImageNet-1k by fine-tuning only 0.1% of its parameters.
Limitations and Future Work
- Sample Efficiency: While AIM scales beautifully, it is less sample-efficient than Masked Autoencoders (MAE). It requires massive amounts of data to reach its full potential.
- Resolution: The current study focuses on fixed-resolution patches. Exploring how autoregressive models handle multi-scale or high-resolution inputs is a natural next step.
- Traversal Patterns: While raster order works best currently, the authors note that the "difficulty" of prediction varies across the image. Future work could explore more sophisticated ways to order or group patches for prediction.
- Beyond Generative: The authors suggest that the synergy between autoregressive objectives and Transformers might eventually lead to a unified architecture for both vision and language.
Summary updated