A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval

Zhixiong Zeng, Wenji MaoarXiv 2022 · OpenReview

Abstract

Cross-Modal Retrieval (CMR) is an important research topic across multimodal computing and information retrieval, which takes one type of data as the query to retrieve relevant data of another type. Recently, vision-language pre-trained models like CLIP have demonstrated superiority in learning visual and textual representations. However, their impact on supervised CMR—where data is retrieved based on semantic categories—has been rarely explored. This paper conducts a comprehensive empirical study using CLIP as a representative model. We propose CLIP4CMR (CLIP enhanced network for Cross-Modal Retrieval) to evaluate performance, robustness to modality imbalance, and sensitivity to hyper-parameters. Our experiments show that CLIP4CMR achieves state-of-the-art results, providing significant implications for model design in supervised CMR.

Introduction

Cross-modal retrieval—the ability to find a relevant image using a text query or vice versa—is a cornerstone of modern search engines and social media. Traditionally, this field is split into unsupervised matching (finding the exact pair, like a specific caption for a specific photo) and supervised retrieval (finding items within the same semantic category, like any "dog" image for the word "canine").

While models like CLIP (Contrastive Language-Image Pre-training) have revolutionized unsupervised matching, their impact on supervised retrieval has been under-explored. Existing supervised methods often struggle with "modality imbalance" (where one type of data is more abundant than the other) and are highly sensitive to hyper-parameters like the size of the embedding space. This study introduces CLIP4CMR, the first supervised CMR framework built upon CLIP, demonstrating its unprecedented robustness and performance.

The CLIP4CMR Framework

The proposed CLIP4CMR framework leverages CLIP’s pre-trained visual and textual encoders as a backbone. Because CLIP’s raw features are optimized for general matching rather than specific class-level discrimination, the authors add modality-specific Multi-Layer Perceptrons (MLPs). These layers project CLIP’s features into a shared "common representation space" where items from the same category are pulled together.

The architecture of CLIP4CMR, showing the integration of CLIP encoders with MLPs and the various learning objectives.

The framework allows for a rigorous comparison of different learning objectives:

  • Pair-wise losses: Comparing data points to each other (e.g., Triplet loss).
  • Class-wise losses: Comparing data points to a "proxy" or "prototype" representing a category (e.g., Prototype Contrastive Loss).
  • Hybrid losses: Combining both approaches.

Key Insights and Experimental Results

The model was evaluated on four standard benchmark datasets: Wikipedia, Pascal-Sentence, NUS-WIDE, and the large-scale XmediaNet.

Intra-class Compactness

A key finding is CLIP4CMR's ability to group items of the same class much more tightly than previous methods. Visualization of distance distributions shows that CLIP’s prior knowledge provides a massive head start in understanding semantic alignment. As seen in the intra-class distance histograms, CLIP4CMR produces a much more compact distribution of distances for items within the same category compared to baselines like PAN.

Comparison of intra-class distance distributions on the Wikipedia dataset between CLIP4CMR and the PAN baseline.

Robustness and Efficiency

  • The "Hybrid Loss" Myth: While previous research suggested combining pair-wise and class-wise losses was essential, this study finds that with a powerful backbone like CLIP, class-wise loss alone (specifically Prototype Contrastive Loss) is often sufficient and sometimes superior.
  • Extreme Robustness: CLIP4CMR is remarkably resilient to missing data. Even when training with only 10% of available text samples, it maintains high performance, whereas traditional baselines see their accuracy plummet.
  • Dimension Insensitivity: Usually, reducing the size of an embedding hurts accuracy. CLIP4CMR is remarkably stable; it performs almost as well with a tiny 64-dimensional embedding as it does with a 1024-dimensional one, which is crucial for practical storage considerations.

Performance and Conclusion

CLIP4CMR achieved new State-of-the-Art (SOTA) results across all four datasets. On the Wikipedia dataset, it improved the mean Average Precision (mAP) by 9.4% over the previous best one-stage method (PAN). On the large-scale XmediaNet, it saw an 8.7% improvement.

While the model shows some sensitivity to numerical stability at high scaling factors, it serves as a robust fundamental framework for future supervised CMR research. Future work may explore incorporating fine-grained data-to-data relations and extending the study to video or audio modalities.

Summary updated

Abstract

Cross-Modal Retrieval (CMR) is an important research topic across multimodal computing and information retrieval, which takes one type of data as the query to retrieve relevant data of another type. It has been widely used in many real-world applications. Recently, the vision-language pre-trained models represented by CLIP demonstrate its superiority in learning the visual and textual representations and gain impressive performance on various vision and language related tasks. Although CLIP as well as the previous pre-trained models have shown great performance improvement in the unsupervised CMR, the performance and impact of these pre-trained models on the supervised CMR were rarely explored due to the lack of common representation for the multimodal class-level associations. In this paper, we take CLIP as the current representative vision-language pre-trained model to conduct a comprehensive empirical study. We evaluate its performance and impact on the supervised CMR, and attempt to answer several key research questions. To this end, we first propose a novel model CLIP4CMR (CLIP enhanced network for Cross-Modal Retrieval) that employs the pre-trained CLIP as backbone network to perform the supervised CMR. Then by means of the CLIP4CMR framework, we revisit the design of different learning objectives in current CMR methods to provide new insights on model design. Moreover, we investigate the most concerned aspects in applying CMR, including the robustness to modality imbalance and sensitivity to hyper-parameters, to provide new perspectives for practical applications. Through extensive experiments, we show that CLIP4CMR achieves the SOTA results with prominent improvements on the benchmark datasets, and can be used as a fundamental framework to empirically study the key research issues of the supervised CMR, with significant implications for model design and practical considerations.
BibTeX
@misc{zeng2022comprehensive,
  author = {Zhixiong Zeng and Wenji Mao},
  title = {{A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval}},
  year = {2022},
  eprint = {2201.02772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.02772},
}
Figure