Contrastive Transformer Cross-Modal Hashing for Video-Text Retrieval

Xiaobo Shen, Qianxin Huang, Long Lan, Yuhui Zheng

Abstract

Contrastive Transformer Cross-Modal Hashing (CTCH) is a novel framework designed to improve the efficiency and accuracy of video-text retrieval. By integrating bidirectional Transformers with a supervised multi-modal contrastive loss, CTCH captures complex temporal dependencies in videos and semantic structures in text, compressing them into compact binary hash codes. This approach facilitates rapid retrieval in a shared Hamming space. Evaluations on benchmark datasets including MSR-VTT, ActivityNet, and Charades show that CTCH consistently outperforms state-of-the-art methods, achieving significant improvements in mean Average Precision (mAP).

Introduction

The rapid growth of video content on social media platforms necessitates advanced retrieval systems. Cross-modal hashing has emerged as a key solution, mapping disparate data types (e.g., video and text) into a common binary space (Hamming space). This allows for near-instantaneous similarity searches. However, traditional hashing methods often fail to capture the sequential nature of video data, either by simple feature averaging or by using computationally intensive LSTMs that struggle with long-term dependencies.

Methodology

The CTCH framework addresses these limitations by employing a Transformer-based architecture designed for high-speed, high-accuracy cross-modal alignment.

Architecture Overview

The CTCH framework architecture, illustrating the flow from data augmentation to the final multi-modal loss functions. Source, p. 2

The CTCH framework architecture, illustrating the flow from data augmentation to the final multi-modal loss functions.
  1. Bidirectional Transformer Encoders: Unlike unidirectional models, CTCH uses bidirectional Transformers to process all frames or words simultaneously. This global view allows the model to capture relationships between distant elements in a sequence, such as the beginning and end of a video clip.
  2. Temporally Consistent Augmentation: To improve robustness without losing temporal structure, the model applies identical augmentations (e.g., cropping, noise) across all frames of a video. Text data is augmented using Easy Data Augmentation (EDA) techniques.
  3. Hash Layer and Quantization: A fully connected layer projects Transformer outputs into a continuous space. These values are then converted into binary hash codes using tanh and sgn functions.
  4. Multi-Modal Supervised Contrastive Loss: This core component utilizes category labels to pull positive pairs (same category) together and push negative pairs apart, both within and across modalities.

Experimental Setup

The performance of CTCH was evaluated on three large-scale video-text benchmarks:

  • MSR-VTT: 10,000 clips across 20 categories with 200,000 sentence pairs.
  • ActivityNet Captions v1.2: Focused on 100 categories of human actions.
  • Charades: Complex daily indoor activities with detailed temporal annotations.

Results and Discussion

CTCH demonstrates superior performance across all tested bit lengths (16, 32, 64, and 128 bits).

Mean Average Precision (mAP) comparison of CTCH against state-of-the-art methods across three datasets and various bit lengths.

Key Findings

  • Performance Gains: On the MSR-VTT dataset, CTCH improved mAP by approximately 3% to 4% over the strongest baselines (e.g., NPH and BTH).
  • Transformer Advantage: The parallel processing of Transformers proved more effective than LSTMs at capturing the global context necessary for generating high-quality, condensed hash codes.
  • Supervised Contrastive Learning: Incorporating explicit label information into the contrastive loss significantly boosted retrieval accuracy compared to purely self-supervised approaches.
  • Ablation Analysis: Removing the inter-modal contrastive loss led to a substantial performance decline, highlighting its role in aligning the video and text representations.

Conclusion

CTCH provides a robust and efficient solution for video-text retrieval by leveraging the strengths of Transformers and supervised contrastive learning. While the model currently relies on high-quality labels and significant initial compute for encoding, it sets a new benchmark for cross-modal hashing. Future research will focus on handling incomplete or noisy multi-modal data, which is prevalent in large-scale, uncurated web environments.

Summary updated

BibTeX
@inproceedings{shen2024contrastive,
  author = {Xiaobo Shen and Qianxin Huang and Long Lan and Yuhui Zheng},
  title = {{Contrastive Transformer Cross-Modal Hashing for Video-Text Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure