Multimodal Contrastive Training for Visual Representation Learning

Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, Baldo Faieta

Abstract

Learning robust visual representations typically requires either massive labeled datasets or extensive self-supervised training on unlabeled images. This paper introduces a unified multimodal contrastive training framework that bridges these approaches by simultaneously exploiting the intrinsic structure of individual modalities and their semantic correlations. By combining self-supervised learning with image-caption and tag-based alignment, the proposed model achieves state-of-the-art performance, including 55.3% top-1 accuracy on ImageNet linear classification using only the COCO dataset for pre-training.

Introduction

Visual representation learning has traditionally been divided into supervised pre-training on large-scale labeled datasets (e.g., ImageNet) and self-supervised learning (SSL) from unlabeled data. While SSL avoids the need for manual labels, it often requires massive datasets or prolonged training to capture high-level semantics. Recently, vision-and-language methods have emerged, using natural language captions to guide visual learning. However, these methods typically focus on cross-modal mapping, neglecting the rich information inherent within each modality.

This paper addresses this limitation by proposing a framework that fully utilizes multimodal data. The goal is to preserve both intra-modal similarity (e.g., consistency between augmented versions of an image) and inter-modal similarity (e.g., alignment between an image and its textual description).

Comparison of different supervision methods for visual representation learning, highlighting the multi-path approach of the proposed method.

Methodology

The proposed multi-path training framework is governed by five distinct contrastive losses, utilizing a ResNet-50 backbone for images and a BERT-like Transformer for text.

Intra-modal Contrastive Learning

To capture the intrinsic structure of each modality, the framework employs three losses:

  1. Image Self-Supervised Loss ($J_{ii}$): Following a MoCo-v2 style approach, this loss ensures that different augmentations of the same image result in similar feature representations.
  2. Text Self-Supervised Loss ($J_{cc}$): This applies a similar logic to captions, using back-translation as data augmentation to maintain semantic consistency in the text embedding space.
  3. Tag Supervised Loss ($J_{tag}$): This novel loss uses image tags to group semantically similar images in the training queue, moving beyond simple instance-level discrimination to categorical alignment.

Inter-modal Contrastive Learning

To align visual and textual concepts, the framework uses two ranking-based contrastive losses:

  1. Image-to-Caption Loss ($J_{ic}$): Ensures a query image is closer to its ground-truth caption than to others in the batch.
  2. Caption-to-Image Loss ($J_{ci}$): Ensures a query caption is closer to its corresponding image.
Visualization of the five contrastive losses used in the proposed framework, including intra-modal and inter-modal objectives.

Decoupled Embedding Spaces

A key architectural innovation is the use of separate MLP heads for intra-modal and inter-modal tasks. This decoupling prevents the different objectives from interfering with each other, allowing the shared backbone to learn features that satisfy both structural consistency and semantic alignment.

Experimental Results

The model was pre-trained on the Microsoft COCO dataset (123K images) and an internal Adobe Stock dataset (5.8M images).

ImageNet Linear Probing

When pre-trained on COCO, the model achieved 55.3% top-1 accuracy on ImageNet. This significantly outperforms previous text-supervised methods like VirTex (52.8%) and standard SSL methods like MoCo-v2 (49.3%), demonstrating the efficiency of the multimodal approach.

Downstream Tasks

On PASCAL VOC object detection, the model reached 81.8 $AP_{50}$, performing competitively with models pre-trained on the much larger ImageNet-1K dataset. Furthermore, the model showed superior performance in cross-modal retrieval tasks, confirming that the learned visual features are well-aligned with natural language.

Conclusion

This research demonstrates that self-supervision and cross-modal supervision are highly complementary. While SSL captures fine-grained visual textures, captions and tags provide high-level abstract concepts. By integrating these signals into a symmetric, multi-path framework, the model achieves high data efficiency and learns more generic visual representations. Future work may extend this framework to other modalities such as video and audio.

Summary updated

Abstract

We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of $55.3\%$ on ImageNet classification, under the common transfer protocol. We also evaluate our method on the large-scale Stock images dataset and show its effectiveness on multi-label image tagging, and cross-modal retrieval tasks.
BibTeX
@inproceedings{yuan2021multimodal,
  author = {Xin Yuan and Zhe Lin and Jason Kuen and Jianming Zhang and Yilin Wang and Michael Maire and Ajinkya Kale and Baldo Faieta},
  title = {{Multimodal Contrastive Training for Visual Representation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00692},
}
Figure