Introduction
Vision-Language Pretraining (VLP) has traditionally relied on the "BERT recipe," where models like LXMERT or UNITER are trained to predict masked words or image regions. However, the standard approach to image region prediction—relying on raw feature regression or object labels from a pre-trained detector—suffers from significant drawbacks. Object labels are often noisy or incorrect, and simple category labels (e.g., "cat") fail to capture the rich, scene-specific semantic nuances essential for complex reasoning tasks.
The DCVLP Framework
To address these limitations, the authors propose Dense Contrastive Visual-Linguistic Pretraining (DCVLP). This method shifts the paradigm from label-dependent classification to self-supervised region-level contrastive learning.
RoI-Feature Dense Contrastive Loss
Instead of predicting a discrete label, DCVLP employs a dual-stream architecture consisting of a QueryNet and a KeyNet. For any given Region of Interest (RoI), the model aims to maximize the similarity between its representation in the QueryNet and its corresponding representation in the KeyNet (the positive pair). Simultaneously, it pushes away representations of all other regions within the same training batch (negative pairs). This contrastive objective allows the model to learn dense, context-aware semantic features without being constrained by a fixed vocabulary of object categories.
Advanced Data Augmentation and Adversarial Training
Since VLP models typically operate on pre-extracted features, standard image augmentations are inapplicable. DCVLP introduces:
- Mask Perturbation: A symmetric masking strategy where text is masked in the QueryNet while image regions are masked in the KeyNet, forcing the model to leverage cross-modal context.
- Adversarial Perturbation: Using Projected Gradient Descent (PGD), the authors inject "calculated noise" into the feature space. This includes Intra-Adversarial noise targeting the contrastive loss and Inter-Adversarial noise generated via the Masked Language Model (MLM) loss to enhance cross-branch robustness.
Experimental Evaluation
The model was pretrained on a composite dataset including MSCOCO, Visual Genome, Conceptual Captions, and SBU Captions. Evaluation across benchmarks like VQA v2.0, GQA, NLVR2, and RefCOCO demonstrated the superiority of the contrastive approach.
Key results include:
- LXMERT Backbone: DCVLP achieved a +2.31 gain on NLVR2 and +0.65 on VQA.
- UNITER Backbone: DCVLP-MASK outperformed the base UNITER model by +0.84 on VQA, proving competitive even against more computationally intensive adversarial frameworks like VILLA.
Conclusion
DCVLP demonstrates that self-supervised contrastive learning provides a more robust and semantically rich foundation for VLP than traditional supervised classification. By eliminating the reliance on noisy object labels and incorporating sophisticated adversarial perturbations, DCVLP sets a new standard for cross-modal representation learning. Future work aims to move beyond frozen object detectors toward fully end-to-end vision-language architectures.
Summary updated