DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever

Zhichao Yin, Binyuan Hui, Min Yang, Fei Huang, Yongbin Li

Abstract

Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the discrepancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves state-of-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.

Introduction

Multi-modal dialog systems, which process both text and images, are becoming increasingly central to human-computer interaction. While generative models like GPT-4 have gained significant attention, retrieval-based systems remain essential for applications requiring high precision, security, and human-verified responses. However, adapting state-of-the-art vision-language models like CLIP (Contrastive Language-Image Pre-training) for dialog tasks presents unique challenges.

Standard CLIP is primarily designed for single image-caption pairs, making it "context-blind" to the flow of multi-turn conversations. Furthermore, there is a significant domain gap between CLIP's training data (static captions) and the informal, interactive nature of human dialog. To address these issues, we present DialCLIP, a parameter-efficient method that transforms CLIP into a high-performance multi-modal dialog retriever by tuning only 0.04% of its total parameters.

Methodology

DialCLIP introduces three innovative components to adapt the frozen CLIP backbone to the complexities of multi-modal dialog without changing its core weights:

1. Multi-Modal Context Prompt Generator (CPG)

Standard CLIP lacks a mechanism to process conversation history. The CPG module takes the entire multi-modal dialog history and distills it into a set of continuous prompt tokens. These tokens are then fed into CLIP’s encoders, providing the model with a "memory" of the preceding turns.

2. Domain Prompts

Dialog data is stylistically distinct from the descriptive captions used to train CLIP. To bridge this gap, DialCLIP inserts learnable "domain prompts" into the intermediate layers of the CLIP architecture. This allows the model to adapt to the specific linguistic and visual patterns of conversational datasets.

3. Mixture of Projection (MoP)

Dialog retrieval often involves different types of matching (e.g., text-to-image or text-to-text). Instead of a single projection head, DialCLIP employs a Mixture of Projection (MoP) strategy. Multiple "expert" projection heads are trained to map CLIP outputs into a shared multi-modal representation space, with each expert specializing in a specific retrieval type.

Experimental Results

The efficacy of DialCLIP was evaluated on two major benchmarks: PhotoChat, a dataset for photo-sharing conversations, and MMDialog, the largest existing multi-modal dialog dataset featuring complex, multi-turn interactions.

Despite tuning a mere 0.04% of the total parameters, DialCLIP achieved state-of-the-art (SOTA) performance:

  • PhotoChat: The model improved the aggregate "Sum" metric (Recall@1, 5, and 10) by 17.8 points, rising from 101.5 to 119.3.
  • MMDialog: DialCLIP saw a significant boost in Image Retrieval (IR@1), increasing the score by 12.8 points (from 34.6 to 47.4).

DialCLIP consistently outperformed much larger, fully fine-tuned models such as PaCE and VLMo, demonstrating that targeted prompt-tuning can be more effective and efficient than brute-force fine-tuning.

Key Insights and Analysis

  • Optimal Context Length: Performance improves with longer context prompts but plateaus or declines after 64–96 tokens. Excessive context appears to introduce noise that can degrade retrieval accuracy.
  • Strategic Layer Placement: Experiments revealed that inserting prompts into the very first embedding layer is not always optimal. For the PhotoChat dataset, the best results were achieved by inserting prompts into the second layer, suggesting that the model benefits from initial low-level feature extraction before context is integrated.
  • Pre-training the "Glue": Initializing the domain prompts and projection heads through a pre-training phase on a subset of data significantly enhanced the final performance.

Conclusion and Future Work

DialCLIP demonstrates that large-scale vision-language models can be effectively adapted for complex dialog tasks through parameter-efficient prompt tuning. By addressing context blindness and domain gaps without modifying the core CLIP weights, DialCLIP offers a lightweight yet powerful solution for multi-modal retrieval.

While highly effective, the current framework focuses primarily on retrieval. Future research could explore how these multi-modal prompts might assist in generative dialog tasks. Additionally, while the tuned parameter count is low, the total inference-time memory footprint remains tied to the large backbone models (Flan-T5 and CLIP ViT), suggesting a need for future work in distilling the context generator into smaller architectures.

Summary updated

Abstract

Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the disc repancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves state-of-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.
BibTeX
@inproceedings{yin2024dialclip,
  author = {Zhichao Yin and Binyuan Hui and Min Yang and Fei Huang and Yongbin Li},
  title = {{DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=RHuZGUaYiu},
}
Figure