Abstract
Cross-Modal Retrieval (CMR) is an important research topic across multimodal computing and information retrieval, which takes one type of data as the query to retrieve relevant data of another type. Recently, vision-language pre-trained models like CLIP have demonstrated superiority in learning visual and textual representations. However, their impact on supervised CMR—where data is retrieved based on semantic categories—has been rarely explored. This paper conducts a comprehensive empirical study using CLIP as a representative model. We propose CLIP4CMR (CLIP enhanced network for Cross-Modal Retrieval) to evaluate performance, robustness to modality imbalance, and sensitivity to hyper-parameters. Our experiments show that CLIP4CMR achieves state-of-the-art results, providing significant implications for model design in supervised CMR.
Introduction
Cross-modal retrieval—the ability to find a relevant image using a text query or vice versa—is a cornerstone of modern search engines and social media. Traditionally, this field is split into unsupervised matching (finding the exact pair, like a specific caption for a specific photo) and supervised retrieval (finding items within the same semantic category, like any "dog" image for the word "canine").
While models like CLIP (Contrastive Language-Image Pre-training) have revolutionized unsupervised matching, their impact on supervised retrieval has been under-explored. Existing supervised methods often struggle with "modality imbalance" (where one type of data is more abundant than the other) and are highly sensitive to hyper-parameters like the size of the embedding space. This study introduces CLIP4CMR, the first supervised CMR framework built upon CLIP, demonstrating its unprecedented robustness and performance.
The CLIP4CMR Framework
The proposed CLIP4CMR framework leverages CLIP’s pre-trained visual and textual encoders as a backbone. Because CLIP’s raw features are optimized for general matching rather than specific class-level discrimination, the authors add modality-specific Multi-Layer Perceptrons (MLPs). These layers project CLIP’s features into a shared "common representation space" where items from the same category are pulled together.
The framework allows for a rigorous comparison of different learning objectives:
- Pair-wise losses: Comparing data points to each other (e.g., Triplet loss).
- Class-wise losses: Comparing data points to a "proxy" or "prototype" representing a category (e.g., Prototype Contrastive Loss).
- Hybrid losses: Combining both approaches.
Key Insights and Experimental Results
The model was evaluated on four standard benchmark datasets: Wikipedia, Pascal-Sentence, NUS-WIDE, and the large-scale XmediaNet.
Intra-class Compactness
A key finding is CLIP4CMR's ability to group items of the same class much more tightly than previous methods. Visualization of distance distributions shows that CLIP’s prior knowledge provides a massive head start in understanding semantic alignment. As seen in the intra-class distance histograms, CLIP4CMR produces a much more compact distribution of distances for items within the same category compared to baselines like PAN.
Robustness and Efficiency
- The "Hybrid Loss" Myth: While previous research suggested combining pair-wise and class-wise losses was essential, this study finds that with a powerful backbone like CLIP, class-wise loss alone (specifically Prototype Contrastive Loss) is often sufficient and sometimes superior.
- Extreme Robustness: CLIP4CMR is remarkably resilient to missing data. Even when training with only 10% of available text samples, it maintains high performance, whereas traditional baselines see their accuracy plummet.
- Dimension Insensitivity: Usually, reducing the size of an embedding hurts accuracy. CLIP4CMR is remarkably stable; it performs almost as well with a tiny 64-dimensional embedding as it does with a 1024-dimensional one, which is crucial for practical storage considerations.
Performance and Conclusion
CLIP4CMR achieved new State-of-the-Art (SOTA) results across all four datasets. On the Wikipedia dataset, it improved the mean Average Precision (mAP) by 9.4% over the previous best one-stage method (PAN). On the large-scale XmediaNet, it saw an 8.7% improvement.
While the model shows some sensitivity to numerical stability at high scaling factors, it serves as a robust fundamental framework for future supervised CMR research. Future work may explore incorporating fine-grained data-to-data relations and extending the study to video or audio modalities.
Summary updated