Contrastive Visual-Linguistic Pretraining

Lei Shi, Kai Shuang, Shijie Geng, Peng Su, Zhengkai Jiang, Peng Gao, Zuohui Fu, Gerard de Melo, Sen SuarXiv 2020 · OpenReview

Abstract

Contrastive Visual-Linguistic Pretraining (CVLP) is a novel self-supervised framework designed to address the limitations of traditional visual feature regression in Vision-Linguistic Pretraining (VLP). By shifting the focus from reconstructing noisy visual features to discriminating between positive and negative image-text pairs, CVLP effectively bridges the "domain gap" between pretraining datasets and real-world applications. Our approach utilizes a dual-network architecture with a dynamic memory queue to provide a large pool of negative samples. Experimental results show that CVLP consistently outperforms existing state-of-the-art models like LXMERT across multiple benchmarks, including VQA v2, GQA, and NLVR2.

Introduction

Recent advancements in Vision-Linguistic Pretraining (VLP) have relied heavily on pretext tasks such as Masked Language Modeling (MLM) and Masked Region Modeling (MRM). While effective, these methods face two significant challenges:

  1. Noisy Labels: Pretraining datasets like Visual Genome often contain incorrect or ambiguous annotations. Forcing a model to regress toward these noisy features leads to the propagation of errors.
  2. Domain Gap: Models trained to predict specific features from a fixed distribution struggle to generalize to "out-of-distribution" images, such as those found in the NLVR2 dataset, which are collected from diverse online sources.

CVLP proposes a paradigm shift: instead of asking "what is in this masked box?", the model is trained to identify "which of these features belongs to this image?" through cross-modality contrastive learning.

Methodology

The CVLP framework is built upon a dual-network architecture consisting of a Query Network and a Key Network.

The CVLP architecture featuring Query and Key networks with a dynamic memory queue for contrastive learning.

Dual-Network and Memory Queue

The Query Network learns via standard backpropagation, while the Key Network maintains a momentum-based moving average of the Query Network's weights. To overcome the memory constraints of Transformers, CVLP employs a Dynamic Memory Queue. This queue stores vision features from previous batches, providing a massive pool of negative samples necessary for effective contrastive learning without the prohibitive computational cost of large batch sizes. Additionally, the authors implement random layer-dropping in the Key Network to prevent overfitting and increase feature diversity.

Training Objectives

CVLP is optimized using a multi-task loss function combining three primary objectives:

  • Masked Language Modeling (MLM): Predicting masked words in the text.
  • Image-Sentence Matching (ISM): Determining if a text describes a given image.
  • Cross-Modality Contrastive Loss: The core innovation, which encourages the model to distinguish correct visual features from a set of negatives stored in the memory queue.

Experimental Results

CVLP was evaluated on three major downstream tasks: Visual Question Answering (VQA v2), GQA (fine-grained reasoning), and Natural Language Visual Reasoning (NLVR2).

Performance comparison between LXMERT and CVLP across VQA, NLVR2, and GQA datasets.

The model demonstrated superior performance across all benchmarks. Notably, on NLVR2, CVLP achieved an accuracy of 76.81%, a significant 2.36% improvement over the LXMERT baseline. This gain on NLVR2—a dataset known for its domain difference from pretraining data—validates CVLP's robustness and generalization capabilities. On VQA v2 and GQA, CVLP reached 73.05% and 61.78% respectively, consistently outperforming previous state-of-the-art models.

Conclusion

CVLP demonstrates that discrimination is often more effective than reconstruction for high-level cross-modal reasoning. By replacing visual feature regression with contrastive learning, CVLP avoids the pitfalls of noisy labels and domain bias inherent in traditional VLP methods. The use of a momentum-updated Key Network and a dynamic memory queue provides a stable and efficient training environment, setting a new standard for Vision-Linguistic Pretraining.

Summary updated

Abstract

Several multi-modality representation learning approaches such as LXMERT and ViLBERT have been proposed recently. Such approaches can achieve superior performance due to the high-level semantic information captured during large-scale multimodal pretraining. However, as ViLBERT and LXMERT adopt visual region regression and classification loss, they often suffer from domain gap and noisy label problems, based on the visual features having been pretrained on the Visual Genome dataset. To overcome these issues, we propose unbiased Contrastive Visual-Linguistic Pretraining (CVLP), which constructs a visual self-supervised loss built upon contrastive learning. We evaluate CVLP on several down-stream tasks, including VQA, GQA and NLVR2 to validate the superiority of contrastive learning on multi-modality representation learning. Our code is available at: https://github.com/ArcherYunDong/CVLP-.
BibTeX
@misc{shi2020contrastive,
  author = {Lei Shi and Kai Shuang and Shijie Geng and Peng Su and Zhengkai Jiang and Peng Gao and Zuohui Fu and Gerard de Melo and Sen Su},
  title = {{Contrastive Visual-Linguistic Pretraining}},
  year = {2020},
  eprint = {2007.13135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.13135},
}
Figure