Abstract
Contrastive Visual-Linguistic Pretraining (CVLP) is a novel self-supervised framework designed to address the limitations of traditional visual feature regression in Vision-Linguistic Pretraining (VLP). By shifting the focus from reconstructing noisy visual features to discriminating between positive and negative image-text pairs, CVLP effectively bridges the "domain gap" between pretraining datasets and real-world applications. Our approach utilizes a dual-network architecture with a dynamic memory queue to provide a large pool of negative samples. Experimental results show that CVLP consistently outperforms existing state-of-the-art models like LXMERT across multiple benchmarks, including VQA v2, GQA, and NLVR2.
Introduction
Recent advancements in Vision-Linguistic Pretraining (VLP) have relied heavily on pretext tasks such as Masked Language Modeling (MLM) and Masked Region Modeling (MRM). While effective, these methods face two significant challenges:
- Noisy Labels: Pretraining datasets like Visual Genome often contain incorrect or ambiguous annotations. Forcing a model to regress toward these noisy features leads to the propagation of errors.
- Domain Gap: Models trained to predict specific features from a fixed distribution struggle to generalize to "out-of-distribution" images, such as those found in the NLVR2 dataset, which are collected from diverse online sources.
CVLP proposes a paradigm shift: instead of asking "what is in this masked box?", the model is trained to identify "which of these features belongs to this image?" through cross-modality contrastive learning.
Methodology
The CVLP framework is built upon a dual-network architecture consisting of a Query Network and a Key Network.
Dual-Network and Memory Queue
The Query Network learns via standard backpropagation, while the Key Network maintains a momentum-based moving average of the Query Network's weights. To overcome the memory constraints of Transformers, CVLP employs a Dynamic Memory Queue. This queue stores vision features from previous batches, providing a massive pool of negative samples necessary for effective contrastive learning without the prohibitive computational cost of large batch sizes. Additionally, the authors implement random layer-dropping in the Key Network to prevent overfitting and increase feature diversity.
Training Objectives
CVLP is optimized using a multi-task loss function combining three primary objectives:
- Masked Language Modeling (MLM): Predicting masked words in the text.
- Image-Sentence Matching (ISM): Determining if a text describes a given image.
- Cross-Modality Contrastive Loss: The core innovation, which encourages the model to distinguish correct visual features from a set of negatives stored in the memory queue.
Experimental Results
CVLP was evaluated on three major downstream tasks: Visual Question Answering (VQA v2), GQA (fine-grained reasoning), and Natural Language Visual Reasoning (NLVR2).
The model demonstrated superior performance across all benchmarks. Notably, on NLVR2, CVLP achieved an accuracy of 76.81%, a significant 2.36% improvement over the LXMERT baseline. This gain on NLVR2—a dataset known for its domain difference from pretraining data—validates CVLP's robustness and generalization capabilities. On VQA v2 and GQA, CVLP reached 73.05% and 61.78% respectively, consistently outperforming previous state-of-the-art models.
Conclusion
CVLP demonstrates that discrimination is often more effective than reconstruction for high-level cross-modal reasoning. By replacing visual feature regression with contrastive learning, CVLP avoids the pitfalls of noisy labels and domain bias inherent in traditional VLP methods. The use of a momentum-updated Key Network and a dynamic memory queue provides a stable and efficient training environment, setting a new standard for Vision-Linguistic Pretraining.
Summary updated