Dense Contrastive Visual-Linguistic Pretraining

Lei Shi, Kai Shuang, Shijie Geng, Peng Gao, Zuohui Fu, Gerard de Melo, Yunpeng Chen, Sen Su

Introduction

Vision-Language Pretraining (VLP) has traditionally relied on the "BERT recipe," where models like LXMERT or UNITER are trained to predict masked words or image regions. However, the standard approach to image region prediction—relying on raw feature regression or object labels from a pre-trained detector—suffers from significant drawbacks. Object labels are often noisy or incorrect, and simple category labels (e.g., "cat") fail to capture the rich, scene-specific semantic nuances essential for complex reasoning tasks.

The DCVLP Framework

To address these limitations, the authors propose Dense Contrastive Visual-Linguistic Pretraining (DCVLP). This method shifts the paradigm from label-dependent classification to self-supervised region-level contrastive learning.

Comparison between traditional masked region loss and the proposed cross-modality region contrastive loss.

RoI-Feature Dense Contrastive Loss

Instead of predicting a discrete label, DCVLP employs a dual-stream architecture consisting of a QueryNet and a KeyNet. For any given Region of Interest (RoI), the model aims to maximize the similarity between its representation in the QueryNet and its corresponding representation in the KeyNet (the positive pair). Simultaneously, it pushes away representations of all other regions within the same training batch (negative pairs). This contrastive objective allows the model to learn dense, context-aware semantic features without being constrained by a fixed vocabulary of object categories.

The DCVLP architecture utilizing QueryNet and KeyNet backbones for RoI-feature contrastive learning.

Advanced Data Augmentation and Adversarial Training

Since VLP models typically operate on pre-extracted features, standard image augmentations are inapplicable. DCVLP introduces:

  • Mask Perturbation: A symmetric masking strategy where text is masked in the QueryNet while image regions are masked in the KeyNet, forcing the model to leverage cross-modal context.
  • Adversarial Perturbation: Using Projected Gradient Descent (PGD), the authors inject "calculated noise" into the feature space. This includes Intra-Adversarial noise targeting the contrastive loss and Inter-Adversarial noise generated via the Masked Language Model (MLM) loss to enhance cross-branch robustness.

Experimental Evaluation

The model was pretrained on a composite dataset including MSCOCO, Visual Genome, Conceptual Captions, and SBU Captions. Evaluation across benchmarks like VQA v2.0, GQA, NLVR2, and RefCOCO demonstrated the superiority of the contrastive approach.

Key results include:

  • LXMERT Backbone: DCVLP achieved a +2.31 gain on NLVR2 and +0.65 on VQA.
  • UNITER Backbone: DCVLP-MASK outperformed the base UNITER model by +0.84 on VQA, proving competitive even against more computationally intensive adversarial frameworks like VILLA.

Conclusion

DCVLP demonstrates that self-supervised contrastive learning provides a more robust and semantically rich foundation for VLP than traditional supervised classification. By eliminating the reliance on noisy object labels and incorporating sophisticated adversarial perturbations, DCVLP sets a new standard for cross-modal representation learning. Future work aims to move beyond frozen object detectors toward fully end-to-end vision-language architectures.

Summary updated

BibTeX
@inproceedings{shi2021dense,
  author = {Lei Shi and Kai Shuang and Shijie Geng and Peng Gao and Zuohui Fu and Gerard de Melo and Yunpeng Chen and Sen Su},
  title = {{Dense Contrastive Visual-Linguistic Pretraining}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475637},
}
Figure