Multimodal Autoregressive Pre-training of Large Vision Encoders

Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, Alexander T. Toshev, Marcin Eichner, Moin Nabi, Yinfei Yang, Joshua M. Susskind, Alaaeldin El-Nouby

Abstract

Recent advances in Large Language Models (LLMs) have demonstrated the power of autoregressive pre-training at scale. AIMV2 extends this success to the multimodal domain, introducing a family of large-scale vision encoders trained to predict both raw image patches and text tokens. By combining a prefix vision encoder with a causal multimodal decoder, AIMV2 achieves state-of-the-art performance, including 89.5% frozen-trunk accuracy on ImageNet-1k, while maintaining the predictable scaling properties characteristic of generative models.

Introduction

Computer vision pre-training has traditionally been split between discriminative/contrastive methods (e.g., CLIP) and generative reconstruction methods (e.g., MAE). While contrastive models are parameter-efficient, they are often difficult to scale due to engineering complexities and massive batch size requirements. Conversely, generative models scale predictably but have historically lagged in downstream performance.

AIMV2 bridges this gap by adopting an autoregressive framework that leverages the strengths of both paradigms: the scalability of generative models and the high performance of discriminative encoders across classification, detection, and multimodal understanding tasks.

Methodology

The AIMV2 architecture is designed to process images and text as a unified sequence. The process begins by "patchifying" the input image, which is then processed through a specialized pipeline.

The AIMV2 architecture featuring a prefix vision encoder and a causal multimodal decoder.

Architectural Components

  1. Prefix Vision Encoder: A Vision Transformer (ViT) that utilizes prefix attention. This allows image patches to attend to each other bidirectionally, facilitating robust feature extraction suitable for downstream tasks without architectural modification.
  2. Causal Multimodal Decoder: A Transformer decoder that processes the concatenated sequence of encoded image features and text embeddings.
  3. Dual Training Objective: The model is trained using a composite loss:
  4. Pixel MSE Loss: For image patches, the model performs pixel-level regression to reconstruct the original visual data.
  5. Cross-entropy Loss: For text, the model performs next-token prediction to generate corresponding captions.

This unified factorization treats the image as a prefix to the text, ensuring that text generation is conditioned on visual features, providing a dense supervisory signal across both modalities.

Datasets and Scaling

AIMV2 was pre-trained on approximately 12 billion image-text samples. This massive dataset includes public sources like DFN-2B and COYO-700M, alongside proprietary high-quality image-text pairs (HQITP) containing both original alt-text and synthetic captions.

The training process demonstrated that LLM-style scaling laws apply to this multimodal framework. As parameter counts and data volume increased, validation loss decreased predictably, offering a more stable training trajectory compared to the often finicky nature of contrastive alternatives at extreme scales.

Key Results and Insights

The experimental results highlight several critical advantages of the AIMV2 approach:

  • State-of-the-Art Recognition: The largest model (AIMV2-3B) achieved a remarkable 89.5% top-1 accuracy on ImageNet-1k using a frozen trunk, surpassing other major self-supervised models like DINOv2.
  • Multimodal Performance: When integrated with a Llama-3 language backbone, AIMV2-L outperformed SigLIP and OpenAI CLIP across various benchmarks, showing particular strength in text-heavy visual tasks like TextVQA and DocVQA.
  • Spatial Understanding: Beyond simple classification, AIMV2 showed superior performance in open-vocabulary object detection and referring expression comprehension, indicating a deep understanding of spatial relationships.
  • Efficiency: The model reached competitive performance levels using significantly fewer samples than contrastive models, demonstrating high data efficiency.

Conclusion and Future Work

AIMV2 proves that autoregressive pre-training is a viable and highly effective strategy for large-scale vision encoders. While the computational cost of pixel-level regression is higher than contrastive losses, the resulting models are more stable to train and exhibit superior downstream performance. Future research will focus on scaling beyond 3 billion parameters and extending the framework to include additional modalities such as video and audio.

Summary updated

BibTeX
@inproceedings{fini2025multimodal,
  author = {Enrico Fini and Mustafa Shukor and Xiujun Li and Philipp Dufter and Michal Klein and David Haldimann and Sai Aitharaju and Victor Guilherme Turrisi da Costa and Louis Béthune and Zhe Gan and Alexander T. Toshev and Marcin Eichner and Moin Nabi and Yinfei Yang and Joshua M. Susskind and Alaaeldin El-Nouby},
  title = {{Multimodal Autoregressive Pre-training of Large Vision Encoders}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00901},
}
Figure