Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions With Multi-Level Representations

Jie Jiang, Shaobo Min, Weijie Kong, Hongfa Wang, Zhifeng Li, Wei Liu

Abstract

Text-Video Retrieval (TVR) is a critical task in cross-modal understanding, requiring the alignment of temporal visual sequences with natural language descriptions. While global contrastive learning methods have shown promise, they often overlook fine-grained semantic interactions and suffer from noisy labels in large-scale training. This paper presents the Hierarchical Cross-Modal Interaction (HCMI) framework, which decomposes the retrieval task into three levels of granularity: frame-word, clip-phrase, and video-sentence. By integrating an adaptive label denoising strategy and marginal sample enhancement, HCMI effectively addresses the "false negative" problem and improves discriminative power. Experimental results on five major benchmarks—MSR-VTT, MSVD, LSMDC, DiDemo, and ActivityNet—demonstrate that HCMI achieves state-of-the-art performance.

Introduction

The explosion of video content has made efficient Text-Video Retrieval (TVR) increasingly important. Unlike image-text retrieval, video retrieval must account for complex temporal dynamics. Most current approaches utilize pre-trained models like CLIP to extract global features, matching a whole video to a whole sentence. However, this "forest-level" approach ignores the "trees"—the specific interactions where individual words correspond to specific frames or phrases correspond to temporal clips. Furthermore, standard contrastive learning assumes that any two different samples in a batch are negative pairs. In reality, multiple videos in a batch may share similar semantic content, leading to "false negatives" that provide noisy gradients during training.

Methodology

The HCMI framework is built upon three core components designed to capture multi-level semantics and improve training robustness.

Hierarchical Representations and Multi-Level Interaction

Instead of relying solely on global vectors, HCMI constructs a hierarchy for both modalities. On the visual side, frames are aggregated into "clips" using self-attention, which are then further fused into a global video representation. Similarly, text is processed from words into "phrases" and finally into a complete sentence representation.

The HCMI framework models interactions at frame-word, clip-phrase, and video-sentence levels.

The model calculates similarity scores and applies contrastive losses at all three levels simultaneously. This ensures that the model learns to align fine-grained details (like specific objects or actions) as well as the overall context.

Adaptive Label Denoising and Marginal Enhancement

To address the limitations of standard contrastive learning, HCMI introduces two key training strategies:

  1. Adaptive Label Denoising: This module identifies videos within a training batch that are semantically similar to the target query. By adjusting the ground-truth labels to account for these similarities, the model avoids treating true semantic matches as negative samples, thereby reducing noise.
  2. Marginal Sample Enhancement: A triplet loss is employed to specifically target "hard negatives"—samples that are deceptively similar to the target. This pushes these samples further away in the feature space, sharpening the model's ability to distinguish between subtle differences.
The training framework utilizes adaptive label denoising to handle false negatives and marginal sample enhancement for hard negatives.

Experimental Results

The HCMI framework was evaluated on five industry-standard benchmarks. Leveraging pre-trained CLIP (ViT-B/16 or ViT-L/14) backbones, the model set new state-of-the-art (SOTA) records across the board.

The Rank@1 (R@1) metrics, representing the percentage of queries where the correct video was the top result, are as follows:

  • MSR-VTT: 55.0% (a significant improvement over previous models like CLIP4Clip)
  • MSVD: 58.2%
  • LSMDC: 29.7%
  • DiDemo: 52.1%
  • ActivityNet: 57.3%

Qualitative analysis via heatmaps confirmed that the model accurately aligns specific nouns and verbs with the corresponding frames and clips where those entities or actions occur.

Conclusion

The HCMI framework demonstrates that hierarchical granularity is essential for effective text-video retrieval. By decomposing the interaction into frame-word, clip-phrase, and video-sentence levels, and addressing the false negative problem through adaptive denoising, the model achieves superior alignment. While the clip-phrase matching remains an area for further refinement, the overall success of HCMI highlights the power of multi-level semantic modeling in cross-modal tasks.

Summary updated

BibTeX
@inproceedings{jiang2025tencent,
  author = {Jie Jiang and Shaobo Min and Weijie Kong and Hongfa Wang and Zhifeng Li and Wei Liu},
  title = {{Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions With Multi-Level Representations}},
  booktitle = {IEEE Access},
  year = {2025},
  url = {https://openreview.net/forum?id=ggm9f7HhhE},
}
Figure