Introduction
mPLUG introduces a novel vision-language architecture featuring cross-modal skip-connections to address the inherent information asymmetry between detailed images and abstract text. By using an asymmetric co-attention mechanism, it achieves state-of-the-art performance across diverse benchmarks while running up to 4x faster than traditional fusion networks.
The Information Asymmetry Problem
In Vision-Language (VL) pre-training, researchers face a significant challenge in balancing information density. Most models utilize one of two primary strategies:
- Connected-attention: Concatenating visual and textual features into a single long sequence.
- Co-attention: Using symmetric interaction between modalities.
However, these approaches suffer from two major drawbacks. First, Information Vanishing occurs because images are represented as long sequences of fine-grained patches, while captions are short and abstract. Dense visual signals can "overwhelm" sparse linguistic signals, leading the model to ignore textual nuances. Second, Computational Inefficiency arises from the quadratic cost of performing full self-attention on long visual sequences, especially as image resolutions increase.
Methods: Cross-modal Skip-connections
The authors propose mPLUG, which utilizes a unique "skip-connected" fusion architecture. Instead of treating both modalities symmetrically at every layer, mPLUG introduces two key structural innovations:
- Asymmetric Co-attention: In the initial layers of a fusion block, the model performs co-attention only on the language side (using visual features as context) and skips the computationally heavy co-attention on the vision side. This preserves the linguistic signal and drastically reduces the number of operations.
- Cross-modal Skip-connections: The model uses a "stride" ($S$) to skip several layers of visual co-attention before performing a single "connected-attention" layer where both modalities are fully fused. This allows fusion to occur at disparate levels of the abstraction hierarchy.
The model is pre-trained end-to-end using four objectives: Image-Text Contrastive (ITC), Image-Text Matching (ITM), Masked Language Modeling (MLM), and Prefix Language Modeling (PrefixLM). This combination enables mPLUG to excel at both discriminative understanding and generative tasks.
Datasets and Evaluation
mPLUG was pre-trained on a collection of 14 million image-text pairs, comprising in-domain data (MS COCO, Visual Genome) and web-scale data (Conceptual Captions, SBU Captions). Evaluation spanned a wide array of downstream tasks, including VQAv2, MS COCO (Captioning and Retrieval), NoCaps, Flickr30K, RefCOCO/+/g (Visual Grounding), NLVR2, and SNLI-VE. Zero-shot transfer was also tested on video-language tasks like MSRVTT and MSVD.
Key Insights
- Asymmetry as an Advantage: Updating visual representations based on text as frequently as text is updated based on visuals is unnecessary. Focusing computation on the more abstract textual side improves both efficiency and signal preservation.
- Efficiency Drives Performance: By skipping redundant visual computations, mPLUG scales to higher image resolutions (e.g., 576x576) without the linguistic signal vanishing, even when visual sequences are 60 times longer than the text.
- Balanced Capabilities: The inclusion of PrefixLM allows mPLUG to function as a versatile foundation model, capable of both precise grounding and high-quality captioning.
Key Results
- Speed: The mPLUG fusion module is 4x faster than conventional connected-attention models, reducing latency from ~1650ms to ~410ms.
- State-of-the-Art Performance: mPLUG achieved SOTA results on VQAv2 (81.27) and MS COCO Captioning (155.1 CIDEr), outperforming models like SimVLM and Florence despite using significantly less training data.
- Zero-Shot Transfer: mPLUG demonstrated remarkable zero-shot performance on video-text retrieval, surpassing models specifically pre-trained on video data.
- Visual Grounding: A +3.16% absolute improvement on RefCOCO testB highlights its superior ability to handle complex spatial queries.
Conclusion
mPLUG effectively addresses the information density gap in vision-language learning through its asymmetric skip-connected architecture. By prioritizing efficiency and signal preservation, it sets a new standard for both performance and speed in multimodal pre-training. Future work may explore scaling to larger datasets and optimizing custom visual encoders for this architecture.
Summary updated