Abstract
Cross-modal hashing is essential for large-scale video-text retrieval, yet existing methods often fail to capture the complex temporal dependencies inherent in video data. We present Contrastive Transformer Cross-Modal Hashing (CTCH), a novel framework that leverages bidirectional transformers to model long-term temporal context in both video and text modalities. Central to our approach is a supervised multi-modality contrastive loss that aligns features within and across modalities using category labels. Our experiments on MSR-VTT, ActivityNet, and Charades benchmarks demonstrate that CTCH significantly outperforms state-of-the-art hashing methods, achieving substantial improvements in mean Average Precision (mAP) while maintaining high retrieval efficiency in Hamming space.
Introduction
The explosion of video content on social media and digital platforms has created an urgent need for efficient cross-modal retrieval systems—tools that allow users to find specific videos using natural language queries. Cross-modal hashing has emerged as the standard solution for large-scale search because it compresses high-dimensional data into compact binary codes. This enables near-instantaneous searching in "Hamming space" and massive savings in storage.
However, many existing hashing methods suffer from a "temporal blind spot." They often treat videos as simple collections of frames, ignoring the sequential relationships that define a video's semantic meaning. While some researchers have employed Recurrent Neural Networks (RNNs) or LSTMs, these models are computationally intensive and often struggle to maintain context over long clips. CTCH addresses these challenges by introducing a transformer-based architecture specifically designed for temporal awareness and cross-modal alignment.
Methodology
Bidirectional Transformer Encoder
At the heart of CTCH is a Bidirectional Transformer Encoder. Unlike standard unidirectional models, this architecture processes video frames and text tokens in both directions simultaneously. This allows the model to understand the context of any given frame or word based on the entire sequence, capturing the "story" of the video far more effectively than simple feature pooling.
Supervised Multi-modality Contrastive Loss
The core innovation of CTCH is its supervised contrastive learning objective. While traditional contrastive learning is often self-supervised, CTCH utilizes category labels to explicitly guide the alignment process. The model is trained to pull similar videos and texts together (both intra-modality and inter-modality) while pushing dissimilar samples apart in the latent space. This ensures that the resulting hash codes are not only compact but also highly discriminative.
Robust Data Augmentation
To enhance model robustness, the authors implement a "temporally consistent" video augmentation strategy. By applying the same transformations (e.g., cropping or flipping) to every frame in a clip, the model preserves the underlying motion structure. For text, Easy Data Augmentation (EDA) techniques are used to improve generalization without losing semantic meaning.
Experimental Evaluation
Datasets and Benchmarks
We evaluated CTCH on three gold-standard video-text benchmarks:
- MSR-VTT: 10,000 video clips across 20 categories with 200,000 clip-sentence pairs.
- ActivityNet Captions v1.2: A large-scale dataset for human action understanding containing over 13,000 videos.
- Charades: Nearly 10,000 videos of daily indoor activities with complex temporal annotations.
Performance Results
CTCH was compared against eight state-of-the-art methods, including deep hashing models like BTH and NPH. The results, summarized in the figure below, show that CTCH consistently outperforms all baselines across different bit lengths (16 to 128 bits).
On the MSR-VTT dataset, CTCH achieved an mAP improvement of approximately 4% to 9.7% over the best-performing baselines in the Text-to-Video task. Ablation studies highlighted the importance of the inter-modality contrastive loss; its removal led to a dramatic performance drop (e.g., from 0.47 to 0.11 on MSR-VTT), confirming its role as the primary driver of cross-modal alignment.
Conclusion
CTCH represents a significant step forward in efficient video-text retrieval. By combining the contextual power of bidirectional transformers with a supervised contrastive learning framework, it overcomes the temporal limitations of previous hashing methods. The model's stability and superior performance across multiple benchmarks make it a promising candidate for real-world, large-scale multimedia search engines. Future work will explore extending this framework to handle incomplete or noisy data where some modalities may be missing during training.
Summary updated