Distilling Vision-Language Pretraining for Efficient Cross-Modal Retrieval

Young Kyun Jang, Donghyun Kim, Ser-Nam LimarXiv 2024 · OpenReview

TL;DR

This paper introduces Distillation for Cross-Modal Quantization (DCMQ), a novel method that leverages powerful Vision-Language Pre-training (VLP) models to significantly improve the efficiency and accuracy of cross-modal retrieval. DCMQ distills the rich semantic understanding from large VLP models into smaller, efficient hashing models, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Efficiently searching through massive, multi-modal datasets (like images and text) is a growing challenge. "Learning to hash" offers a solution by compressing high-dimensional data into compact binary codes, enabling fast retrieval. However, traditional supervised hashing methods often rely on limited, multi-hot labels that lack rich semantic information. While large pre-trained Vision-Language Pre-training (VLP) models possess deep semantic understanding, directly using them for retrieval is computationally expensive and can lead to a loss of fine-grained understanding when fine-tuned for specific tasks. This paper aims to bridge this gap by efficiently transferring the knowledge from these powerful VLP models to create highly effective and efficient hashing-based cross-modal retrieval systems.

Methods

The core of the proposed method is Distillation for Cross-Modal Quantization (DCMQ). It works by using a large VLP model as a "teacher" to guide a smaller "student" hashing model.

Here's a breakdown of the key techniques:

  • Knowledge Distillation: Instead of using traditional, sparse multi-hot labels, DCMQ uses the VLP model to generate richer, semantically informed target similarity scores between images and text. This is achieved by converting labels into descriptive text and then using the VLP to compute cross-modal similarities.
  • Normalization with Paired Consistency (NPC): A crucial step to make the distilled targets effective. Raw VLP similarity scores can be densely distributed and lack discrimination. NPC transforms these scores to a more discriminative range (-1.0 to 1.0) while preserving the consistency between paired image-text data, ensuring that relevant pairs have higher scores.
  • Product Quantization with Gumbel (PQG): To improve the hashing process itself, the authors introduce PQG. This method uses Gumbel noise during the quantization of embeddings. This regularization encourages a more balanced use of codewords across the codebooks, preventing certain codewords from dominating and thus increasing the model's capacity to capture more nuanced semantics and improve generalization.
  • Joint Training: The image encoder, text encoder, and codebooks of the student model are trained end-to-end to learn a shared embedding space.

Datasets

The proposed method was evaluated on three standard cross-modal retrieval benchmark datasets:

  • MS COCO: A large-scale dataset with images and associated semantic labels, also featuring detailed text captions.
  • MIRFlickr: A dataset of images collected from Flickr with multiple class labels.
  • NUS-WIDE: A real-world web image database with conceptual labels.

The authors also mention using actual text captions from MS COCO for a specific experiment, demonstrating the flexibility of their approach.

Key Insights and Ideas

  • Leveraging VLP for Hashing is Powerful: The core idea that large VLP models can significantly boost the performance of hashing-based retrieval is validated. The rich semantic understanding embedded in VLPs is far more beneficial than traditional multi-hot labels.
  • Raw VLP Scores Need Normalization: Simply using raw VLP similarity scores as distillation targets is suboptimal. The NPC technique is essential for making these targets discriminative and effective for training the hashing model.
  • Gumbel Noise Enhances Codebook Usage: PQG's use of Gumbel noise leads to more balanced codeword selection, which is critical for improving the information capacity and generalization of the hashing system.
  • Teacher Model Matters: The performance of DCMQ is influenced by the choice of VLP teacher. Stronger, more capable VLP models generally lead to better retrieval accuracy, demonstrating the effectiveness of the distillation approach.
  • Efficiency Gains are Significant: DCMQ achieves comparable or better performance than other methods while drastically reducing storage requirements, making it practical for large-scale deployment.

Key Results

  • State-of-the-Art Performance: DCMQ consistently outperforms existing supervised cross-modal hashing approaches across all tested datasets and bit lengths, achieving the best mAP scores.
  • Significant Improvements: For the I → T task, DCMQ shows improvements of 30.3%p, 44.4%p, and 46.6%p over non-deep methods on MIRFlickr, NUS-WIDE, and MS COCO, respectively. When compared to other deep learning methods, DCMQ still achieves notable gains.
  • Efficiency: DCMQ significantly reduces storage costs (over 120 times less than floating-type embeddings) and offers faster search speeds due to its approximate nearest neighbor search capabilities.
  • Robustness to Low Bits: DCMQ demonstrates comparable performance even under extremely low bit conditions (e.g., 8-bit), highlighting its efficiency.
  • Qualitative Success: Visual retrieval examples show DCMQ generating high-quality and relevant results.

Limitations and Future Work

  • Untested in Other Tasks: The paper acknowledges that DCMQ's effectiveness in tasks beyond retrieval remains untested.
  • Potential Social Impacts: The authors briefly mention potential privacy concerns and the risk of discriminatory outcomes due to the reliance on training data, emphasizing the need for careful mitigation measures.
  • Future Work: The paper suggests expanding DCMQ's applicability to other tasks and exploring ways to address potential social impacts. The exploration of different image encoder backbones also hints at future work in optimizing the student model architecture.

Summary updated

Abstract

Learning to hash'' is a practical solution for efficient retrieval, offering fast search speed and low storage cost. It is widely applied in various applications, such as image-text cross-modal search. In this paper, we explore the potential of enhancing the performance of learning to hash with the proliferation of powerful large pre-trained models, such as Vision-Language Pre-training (VLP) models. We introduce a novel method named Distillation for Cross-Modal Quantization (DCMQ), which leverages the rich semantic knowledge of VLP models to improve hash representation learning. Specifically, we use the VLP as a `teacher' to distill knowledge into a `student' hashing model equipped with codebooks. This process involves the replacement of supervised labels, which are composed of multi-hot vectors and lack semantics, with the rich semantics of VLP. In the end, we apply a transformation termed Normalization with Paired Consistency (NPC) to achieve a discriminative target for distillation. Further, we introduce a new quantization method, Product Quantization with Gumbel (PQG) that promotes balanced codebook learning, thereby improving the retrieval performance. Extensive benchmark testing demonstrates that DCMQ consistently outperforms existing supervised cross-modal hashing approaches, showcasing its significant potential.
BibTeX
@misc{jang2024distilling,
  author = {Young Kyun Jang and Donghyun Kim and Ser-Nam Lim},
  title = {{Distilling Vision-Language Pretraining for Efficient Cross-Modal Retrieval}},
  year = {2024},
  eprint = {2405.14726},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.14726},
}
Figure