DialCLIP: Empowering Clip As Multi-Modal Dialog Retriever

Zhichao Yin, Binyuan Hui, Min Yang, Fei Huang, Yongbin Li

TL;DR

DialCLIP is a parameter-efficient method that transforms the popular CLIP model into a high-performance multi-modal dialog retriever. By tuning a mere 0.04% of total parameters using a novel prompt-tuning architecture, the researchers achieved state-of-the-art results on major benchmarks, significantly outperforming models that require full fine-tuning.

What Problem Does This Paper Address?

While Large Language Models (LLMs) like GPT-4 are excellent at generating text, retrieval-based systems remain vital for multi-modal dialogs (conversations involving both text and images). Retrieval systems are often preferred in security-sensitive or privacy-conscious applications because they provide controlled, human-verified responses rather than "hallucinated" ones.

The authors identify three major hurdles when using the industry-standard CLIP (Contrastive Language-Image Pre-training) for this task:

  1. Context Blindness: CLIP is trained on single image-caption pairs and struggles to grasp the "flow" of a multi-turn conversation.
  2. Domain Gap: There is a significant stylistic difference between the descriptive captions CLIP was trained on and the informal, interactive nature of human dialog.
  3. Modality Mismatch: Standard CLIP uses a single projection scheme, which isn't flexible enough to handle the various retrieval types required in a dialog (e.g., retrieving a relevant image vs. a relevant text response).

Methods

The authors introduce DialCLIP, which uses three key components to "re-wire" CLIP without changing its core weights:

  • Context Prompt Generator (CPG): This module takes the entire multi-modal dialog history and "distills" it into a set of continuous prompt tokens. These tokens are then fed into CLIP’s encoders, effectively giving the model a "memory" of the conversation.
  • Domain Prompts: To bridge the gap between static captions and dynamic dialog, the authors insert learnable "domain prompts" into the intermediate layers of CLIP. This helps the model adapt to the specific linguistic and visual patterns of conversational data.
  • Mixture of Projection (MoP): Instead of a single output layer, DialCLIP uses multiple "expert" projection heads. Each expert is responsible for a specific type of retrieval (e.g., text-to-image), allowing the model to map different modalities into a shared representation space more accurately.

Datasets

The model was evaluated on two widely recognized benchmarks:

  • PhotoChat: The first dataset specifically designed for the "photo sharing" task in human-human conversations.
  • MMDialog: Currently the largest multi-modal dialog dataset, featuring complex, multi-turn interactions.

Key Insights and Ideas

  • Extreme Efficiency: The most striking finding is that tuning only 0.04% of the parameters (the prompts and projection heads) yields better results than fine-tuning the entire model. This makes DialCLIP incredibly lightweight for deployment.
  • The "Goldilocks" Context Length: The researchers found that longer context prompts generally improve performance, but only up to a point (around 64–96 tokens). Beyond that, performance drops, likely because excessive context introduces "noise" that confuses the model.
  • Layer Placement Matters: Surprisingly, inserting prompts into the very first (embedding) layer isn't optimal. For the PhotoChat dataset, inserting prompts into the 2nd layer yielded the best results, suggesting that the model needs some initial low-level feature processing before the context is introduced.
  • Pre-training the "Glue": The authors found that "pre-training" the domain prompts and projection heads on a subset of data before the main training phase significantly boosted final performance.

Key Results

DialCLIP set new state-of-the-art (SOTA) benchmarks across the board:

  • PhotoChat: Improved the "Sum" metric (a aggregate of Recall@1, 5, and 10) by 17.8 points (from 101.5 to 119.3).
  • MMDialog: Achieved a significant boost in Image Retrieval (IR@1), increasing the score by 12.8 points (from 34.6 to 47.4).
  • Comparison: It consistently outperformed much larger, fully fine-tuned models like PaCE and VLMo, proving that smart prompt engineering can be more effective than brute-force training.

Limitations and Future Work

While DialCLIP is highly effective for retrieval, the paper focuses primarily on the retrieval task. The authors do not explore how these multi-modal prompts might assist in generative dialog tasks (where the model writes its own response). Additionally, while the parameter count is low, the Context Prompt Generator itself relies on a frozen Flan-T5 and CLIP ViT, meaning the total inference-time memory footprint is still tied to these large backbone models. Future research could look into distilling the CPG into an even smaller architecture.

Summary updated

Abstract

Recently, substantial advancements in pre-trained visionlanguage models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the disc repancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves stateof-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.
BibTeX
@inproceedings{yin2024dialclip,
  author = {Zhichao Yin and Binyuan Hui and Min Yang and Fei Huang and Yongbin Li},
  title = {{DialCLIP: Empowering Clip As Multi-Modal Dialog Retriever}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=RHuZGUaYiu},
}

Loading related content...

Figure