Abstract
Recent advances in Large Language Models (LLMs) have demonstrated the power of autoregressive pre-training at scale. AIMV2 extends this success to the multimodal domain, introducing a family of large-scale vision encoders trained to predict both raw image patches and text tokens. By combining a prefix vision encoder with a causal multimodal decoder, AIMV2 achieves state-of-the-art performance, including 89.5% frozen-trunk accuracy on ImageNet-1k, while maintaining the predictable scaling properties characteristic of generative models.
Introduction
Computer vision pre-training has traditionally been split between discriminative/contrastive methods (e.g., CLIP) and generative reconstruction methods (e.g., MAE). While contrastive models are parameter-efficient, they are often difficult to scale due to engineering complexities and massive batch size requirements. Conversely, generative models scale predictably but have historically lagged in downstream performance.
AIMV2 bridges this gap by adopting an autoregressive framework that leverages the strengths of both paradigms: the scalability of generative models and the high performance of discriminative encoders across classification, detection, and multimodal understanding tasks.
Methodology
The AIMV2 architecture is designed to process images and text as a unified sequence. The process begins by "patchifying" the input image, which is then processed through a specialized pipeline.
Architectural Components
- Prefix Vision Encoder: A Vision Transformer (ViT) that utilizes prefix attention. This allows image patches to attend to each other bidirectionally, facilitating robust feature extraction suitable for downstream tasks without architectural modification.
- Causal Multimodal Decoder: A Transformer decoder that processes the concatenated sequence of encoded image features and text embeddings.
- Dual Training Objective: The model is trained using a composite loss:
- Pixel MSE Loss: For image patches, the model performs pixel-level regression to reconstruct the original visual data.
- Cross-entropy Loss: For text, the model performs next-token prediction to generate corresponding captions.
This unified factorization treats the image as a prefix to the text, ensuring that text generation is conditioned on visual features, providing a dense supervisory signal across both modalities.
Datasets and Scaling
AIMV2 was pre-trained on approximately 12 billion image-text samples. This massive dataset includes public sources like DFN-2B and COYO-700M, alongside proprietary high-quality image-text pairs (HQITP) containing both original alt-text and synthetic captions.
The training process demonstrated that LLM-style scaling laws apply to this multimodal framework. As parameter counts and data volume increased, validation loss decreased predictably, offering a more stable training trajectory compared to the often finicky nature of contrastive alternatives at extreme scales.
Key Results and Insights
The experimental results highlight several critical advantages of the AIMV2 approach:
- State-of-the-Art Recognition: The largest model (AIMV2-3B) achieved a remarkable 89.5% top-1 accuracy on ImageNet-1k using a frozen trunk, surpassing other major self-supervised models like DINOv2.
- Multimodal Performance: When integrated with a Llama-3 language backbone, AIMV2-L outperformed SigLIP and OpenAI CLIP across various benchmarks, showing particular strength in text-heavy visual tasks like TextVQA and DocVQA.
- Spatial Understanding: Beyond simple classification, AIMV2 showed superior performance in open-vocabulary object detection and referring expression comprehension, indicating a deep understanding of spatial relationships.
- Efficiency: The model reached competitive performance levels using significantly fewer samples than contrastive models, demonstrating high data efficiency.
Conclusion and Future Work
AIMV2 proves that autoregressive pre-training is a viable and highly effective strategy for large-scale vision encoders. While the computational cost of pixel-level regression is higher than contrastive losses, the resulting models are more stable to train and exhibit superior downstream performance. Future research will focus on scaling beyond 3 billion parameters and extending the framework to include additional modalities such as video and audio.
Summary updated