Abstract
Learning robust visual representations typically requires either massive labeled datasets or extensive self-supervised training on unlabeled images. This paper introduces a unified multimodal contrastive training framework that bridges these approaches by simultaneously exploiting the intrinsic structure of individual modalities and their semantic correlations. By combining self-supervised learning with image-caption and tag-based alignment, the proposed model achieves state-of-the-art performance, including 55.3% top-1 accuracy on ImageNet linear classification using only the COCO dataset for pre-training.
Introduction
Visual representation learning has traditionally been divided into supervised pre-training on large-scale labeled datasets (e.g., ImageNet) and self-supervised learning (SSL) from unlabeled data. While SSL avoids the need for manual labels, it often requires massive datasets or prolonged training to capture high-level semantics. Recently, vision-and-language methods have emerged, using natural language captions to guide visual learning. However, these methods typically focus on cross-modal mapping, neglecting the rich information inherent within each modality.
This paper addresses this limitation by proposing a framework that fully utilizes multimodal data. The goal is to preserve both intra-modal similarity (e.g., consistency between augmented versions of an image) and inter-modal similarity (e.g., alignment between an image and its textual description).
Methodology
The proposed multi-path training framework is governed by five distinct contrastive losses, utilizing a ResNet-50 backbone for images and a BERT-like Transformer for text.
Intra-modal Contrastive Learning
To capture the intrinsic structure of each modality, the framework employs three losses:
- Image Self-Supervised Loss ($J_{ii}$): Following a MoCo-v2 style approach, this loss ensures that different augmentations of the same image result in similar feature representations.
- Text Self-Supervised Loss ($J_{cc}$): This applies a similar logic to captions, using back-translation as data augmentation to maintain semantic consistency in the text embedding space.
- Tag Supervised Loss ($J_{tag}$): This novel loss uses image tags to group semantically similar images in the training queue, moving beyond simple instance-level discrimination to categorical alignment.
Inter-modal Contrastive Learning
To align visual and textual concepts, the framework uses two ranking-based contrastive losses:
- Image-to-Caption Loss ($J_{ic}$): Ensures a query image is closer to its ground-truth caption than to others in the batch.
- Caption-to-Image Loss ($J_{ci}$): Ensures a query caption is closer to its corresponding image.
Decoupled Embedding Spaces
A key architectural innovation is the use of separate MLP heads for intra-modal and inter-modal tasks. This decoupling prevents the different objectives from interfering with each other, allowing the shared backbone to learn features that satisfy both structural consistency and semantic alignment.
Experimental Results
The model was pre-trained on the Microsoft COCO dataset (123K images) and an internal Adobe Stock dataset (5.8M images).
ImageNet Linear Probing
When pre-trained on COCO, the model achieved 55.3% top-1 accuracy on ImageNet. This significantly outperforms previous text-supervised methods like VirTex (52.8%) and standard SSL methods like MoCo-v2 (49.3%), demonstrating the efficiency of the multimodal approach.
Downstream Tasks
On PASCAL VOC object detection, the model reached 81.8 $AP_{50}$, performing competitively with models pre-trained on the much larger ImageNet-1K dataset. Furthermore, the model showed superior performance in cross-modal retrieval tasks, confirming that the learned visual features are well-aligned with natural language.
Conclusion
This research demonstrates that self-supervision and cross-modal supervision are highly complementary. While SSL captures fine-grained visual textures, captions and tags provide high-level abstract concepts. By integrating these signals into a symmetric, multi-path framework, the model achieves high data efficiency and learns more generic visual representations. Future work may extend this framework to other modalities such as video and audio.
Summary updated