VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

Wenhui Wang, Hangbo Bao, Li Dong, Furu Wei

Abstract

Vision-Language (VL) models typically face a trade-off between the efficiency of dual encoders for retrieval and the accuracy of fusion encoders for reasoning. We present VLMo (Vision-Language Mixture-of-Experts), a unified architecture that bridges this gap using a novel Mixture-of-Modality-Experts (MOME) Transformer. By dynamically switching between modality-specific experts while sharing self-attention parameters, VLMo can act as both a dual and a fusion encoder. Coupled with a stagewise pre-training strategy on unimodal and multimodal data, VLMo achieves state-of-the-art results across various VL benchmarks, including VQA and image-text retrieval.

Introduction

In vision-language pre-training, two architectural paradigms dominate. Dual Encoders (e.g., CLIP) encode images and text separately, enabling fast retrieval through pre-computed embeddings but struggling with complex cross-modal reasoning. Conversely, Fusion Encoders (e.g., ViLT) employ deep cross-modal attention for high-accuracy reasoning but suffer from high computational complexity during retrieval. VLMo addresses this dichotomy by providing a single, unified model flexible enough to handle both paradigms efficiently.

Methods: The Mixture-of-Modality-Experts (MOME) Transformer

The core innovation of VLMo is the Mixture-of-Modality-Experts (MOME) Transformer block. Unlike standard Transformers where every block uses a single Feed-Forward Network (FFN), the MOME block contains a pool of modality-specific experts: a Vision Expert (V-FFN), a Language Expert (L-FFN), and a Vision-Language Expert (VL-FFN).

Overview of the MoME Transformer architecture and its flexible configuration for different vision-language tasks.

All modalities share the same Multi-Head Self-Attention parameters, which forces the model to align different modalities into a common representation space. The model "switches" experts based on the task:

  • Dual Encoder Mode: Uses V-FFN for images and L-FFN for text to enable efficient retrieval (Image-Text Contrastive learning).
  • Fusion Encoder Mode: Uses the VL-FFN in upper layers to model deep interactions for reasoning tasks like VQA or Image-Text Matching.

Stagewise Pre-training

To effectively leverage abundant unimodal data, VLMo employs a three-stage training strategy:

  1. Vision Pre-training: Training vision experts and self-attention on image-only data using masked image modeling (BEiT).
  2. Language Pre-training: Freezing vision components and training language experts on text-only data (Masked Language Modeling).
  3. Vision-Language Pre-training: Training the entire model on image-text pairs using Image-Text Contrastive (ITC), Image-Text Matching (ITM), and Masked Language Modeling (MLM) objectives.
The stagewise pre-training pipeline, transitioning from unimodal vision and language data to multimodal image-text pairs.

Results and Key Insights

VLMo was evaluated on standard benchmarks like VQA v2.0, NLVR2, and image-text retrieval (MSCOCO, Flickr30K). The model demonstrates that FFNs act as "knowledge containers" for modality-specific information, while shared self-attention handles global alignment.

Key results include:

  • State-of-the-Art Performance: VLMo-Large++ reached 82.88% on VQA test-dev and 89.54% on NLVR2 test-P, outperforming significantly larger models.
  • Efficiency: VLMo-Base, trained on only 4M images, outperformed models like ALBEF and ViLT, highlighting the parameter efficiency of the MOME architecture.
  • Retrieval Speed: VLMo maintains the linear inference speed of dual encoders for retrieval while achieving the accuracy of fusion encoders.

Conclusion

VLMo introduces a versatile architecture that unifies retrieval and reasoning capabilities. By utilizing modality-specific experts within a shared attention framework and a robust stagewise pre-training pipeline, it sets a new standard for vision-language models. Future work includes extending the MOME framework to generative tasks and additional modalities like audio or video.

Summary updated

BibTeX
@inproceedings{wang2022vlmo,
  author = {Wenhui Wang and Hangbo Bao and Li Dong and Furu Wei},
  title = {{VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=bydKs84JEyw},
}
Figure