Trust Your Partner's Friends: Hierarchical Cross-Modal Contrastive Pre-Training for Video-Text Retrieval

Yuhan Xiang, Kaijian Liu, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Xianming Lin

Abstract

Video-text retrieval has greatly benefited from the massive web video in recent years, while the performance is still limited to the weak supervision from the uncurated data. In this work, we propose to leverage the well-represented information of each original modality and exploit complementary information in two views of the same video, i.e., video clips and captions, by using one view to obtain positive samples with the neighboring samples of the other. Respecting the hierarchical organization of real-world data, we further design a hierarchical cross-modal pre-training method (HCP) to learn good representations in the common embedding space. We evaluate the pre-trained model on three downstream tasks, i.e. text-to-video retrieval, action step localization and video question answering and our method outperforms previous works under the same setting.
BibTeX
@inproceedings{xiang2023trust,
  author = {Yuhan Xiang and Kaijian Liu and Shixiang Tang and Lei Bai and Feng Zhu and Rui Zhao and Xianming Lin},
  title = {{Trust Your Partner's Friends: Hierarchical Cross-Modal Contrastive Pre-Training for Video-Text Retrieval}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10097061},
}

Loading related content...

Figure