Abstract
Vision-Language (VL) models typically face a trade-off between the efficiency of dual encoders for retrieval and the accuracy of fusion encoders for reasoning. We present VLMo (Vision-Language Mixture-of-Experts), a unified architecture that bridges this gap using a novel Mixture-of-Modality-Experts (MOME) Transformer. By dynamically switching between modality-specific experts while sharing self-attention parameters, VLMo can act as both a dual and a fusion encoder. Coupled with a stagewise pre-training strategy on unimodal and multimodal data, VLMo achieves state-of-the-art results across various VL benchmarks, including VQA and image-text retrieval.
Introduction
In vision-language pre-training, two architectural paradigms dominate. Dual Encoders (e.g., CLIP) encode images and text separately, enabling fast retrieval through pre-computed embeddings but struggling with complex cross-modal reasoning. Conversely, Fusion Encoders (e.g., ViLT) employ deep cross-modal attention for high-accuracy reasoning but suffer from high computational complexity during retrieval. VLMo addresses this dichotomy by providing a single, unified model flexible enough to handle both paradigms efficiently.
Methods: The Mixture-of-Modality-Experts (MOME) Transformer
The core innovation of VLMo is the Mixture-of-Modality-Experts (MOME) Transformer block. Unlike standard Transformers where every block uses a single Feed-Forward Network (FFN), the MOME block contains a pool of modality-specific experts: a Vision Expert (V-FFN), a Language Expert (L-FFN), and a Vision-Language Expert (VL-FFN).
All modalities share the same Multi-Head Self-Attention parameters, which forces the model to align different modalities into a common representation space. The model "switches" experts based on the task:
- Dual Encoder Mode: Uses V-FFN for images and L-FFN for text to enable efficient retrieval (Image-Text Contrastive learning).
- Fusion Encoder Mode: Uses the VL-FFN in upper layers to model deep interactions for reasoning tasks like VQA or Image-Text Matching.
Stagewise Pre-training
To effectively leverage abundant unimodal data, VLMo employs a three-stage training strategy:
- Vision Pre-training: Training vision experts and self-attention on image-only data using masked image modeling (BEiT).
- Language Pre-training: Freezing vision components and training language experts on text-only data (Masked Language Modeling).
- Vision-Language Pre-training: Training the entire model on image-text pairs using Image-Text Contrastive (ITC), Image-Text Matching (ITM), and Masked Language Modeling (MLM) objectives.
Results and Key Insights
VLMo was evaluated on standard benchmarks like VQA v2.0, NLVR2, and image-text retrieval (MSCOCO, Flickr30K). The model demonstrates that FFNs act as "knowledge containers" for modality-specific information, while shared self-attention handles global alignment.
Key results include:
- State-of-the-Art Performance: VLMo-Large++ reached 82.88% on VQA test-dev and 89.54% on NLVR2 test-P, outperforming significantly larger models.
- Efficiency: VLMo-Base, trained on only 4M images, outperformed models like ALBEF and ViLT, highlighting the parameter efficiency of the MOME architecture.
- Retrieval Speed: VLMo maintains the linear inference speed of dual encoders for retrieval while achieving the accuracy of fusion encoders.
Conclusion
VLMo introduces a versatile architecture that unifies retrieval and reasoning capabilities. By utilizing modality-specific experts within a shared attention framework and a robust stagewise pre-training pipeline, it sets a new standard for vision-language models. Future work includes extending the MOME framework to generative tasks and additional modalities like audio or video.
Summary updated