DeCMG: Denoise with Cross-modality Guidance Makes Better Text-Video Retrieval

Rui Deng, Yuke LiICME 2024 · OpenReview

Abstract

The effectiveness of text-video retrieval depends on establishing precise semantic correspondence between text and video. However, current CLIP-based [1] approaches, have encountered challenges in adequately learning temporal and cross-modality features due to the lack of prior knowledge and the scarcity of labeled training data. To tackle this issue, we propose DeCMG (Denoise with Cross-Modality Guidance), a novel method that enhances fine-grained text-video alignment through cross-modality collaboration. The collaboration in DeCMG allows the features in one modality to be better optimized by performing a denoising task in another modality with a guidance vector. Specifically, DeCMG incorporates two self-generated denoising tasks (TextDe and VisualDe) to achieve alignment at the word-video and frame-text levels, respectively. These tasks are implemented as plugins, resulting in only a minor increase in training time and can be entirely removed during inference. Across four benchmark datasets, DeCMG yielded average improvements of 1.1% in text-to-video and 1.8% in video-to-text Recall@1, surpassing our baseline models (X-CLIP [2] and X-Pool [3]). These experimental results validate the efficiency and effectiveness of DeCMG.
BibTeX
@inproceedings{deng2024decmg,
  author = {Rui Deng and Yuke Li},
  title = {{DeCMG: Denoise with Cross-modality Guidance Makes Better Text-Video Retrieval}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=AXnxfDo7CV},
}

Loading related content...

Figure