HunYuan_tvr for Text-Video Retrievial

Shaobo Min, Weijie Kong, Rong-Cheng Tu, Dihong Gong, Chengfei Cai, Wenzhe Zhao, Chenyang Liu, Sixiao Zheng, Hongfa Wang, Zhifeng Li, Wei Liu

Abstract

Text-Video Retrieval plays an important role in multi-modal understanding and has attracted increasing attention in recent years. Most existing methods focus on constructing contrastive pairs between whole videos and complete caption sentences, while ignoring fine-grained cross-modal relationships, e.g., short clips and phrases or single frame and word. In this paper, we propose a novel method, named HunYuan_tvr, to explore hierarchical cross-modal interactions by simultaneously exploring video-sentence, clip-phrase, and frame-word relationships. Considering intrinsic semantic relations between frames, HunYuan_tvr first performs self-attention to explore frame-wise correlations and adaptively clusters correlated frames into clip-level representations. Then, the clip-wise correlation is explored to aggregate clip representations into a compact one to describe the video globally. In this way, we can construct hierarchical video representations for frame-clip-video granularities, and also explore word-wise correlations to form word-phrase-sentence embeddings for the text modality. Finally, hierarchical contrastive learning is designed to explore cross-modal relationships, i.e., frame-word, clip-phrase, and video-sentence, which enables HunYuan_tvr to achieve a comprehensive multi-modal understanding. Further boosted by adaptive label denoising and marginal sample enhancement, HunYuan_tvr obtains new state-of-the-art results on various benchmarks, e.g., Rank@1 of 55.0%, 57.8%, 29.7%, 52.1%, and 57.3% on MSR-VTT, MSVD, LSMDC, DiDemo, and ActivityNet respectively.

Introduction

Text-Video Retrieval (TVR) is a fundamental task in multi-modal understanding, aiming to retrieve relevant videos for a given text query. While models like CLIP have significantly advanced image-text alignment, extending this to video introduces the challenge of temporal dynamics. Current TVR approaches often rely on global matching, which overlooks fine-grained details, or simple token-level matching, which misses high-level semantic context.

Furthermore, standard contrastive learning frameworks often suffer from the "noisy label" problem. In a typical training batch, all samples except the ground truth are treated as negatives. However, multiple videos might share similar semantic content (e.g., "a person running"), leading to false negatives that hinder the model's ability to learn nuanced features.

Hierarchical Cross-Modal Interaction

To address these limitations, we introduce HunYuan_tvr, a framework centered on Hierarchical Cross-Modal Interaction (HCMI). Instead of a single global vector, HCMI constructs a hierarchy for both visual and textual modalities.

The HCMI architecture illustrates the multi-level aggregation from frames/words to clips/phrases and finally to global video/sentence representations.

Multi-Level Representations

The model processes data at three distinct granularities:

  1. Frame-Word Level: Captures fine-grained object-level interactions.
  2. Clip-Phrase Level: Aggregates frames into clips via self-attention to capture short-term actions and matches them with textual phrases.
  3. Video-Sentence Level: Provides a global summary of the entire video and sentence.

The visual hierarchy is built by adaptively clustering correlated frames into clip-level representations, which are then further aggregated into a global video embedding. A similar process is applied to the text modality to form word-phrase-sentence embeddings.

Training Strategy and Optimization

Beyond the hierarchical architecture, HunYuan_tvr employs advanced training strategies to improve robustness and discriminative power.

The training pipeline incorporates label denoising to handle false negatives and hard sample mining to enhance discriminative learning.

Adaptive Label Denoising

To mitigate the false negative problem, we utilize an adaptive label denoising strategy. By comparing different random samplings of the same video against other videos in the batch, the model can identify "potential positives"—videos that are semantically similar to the target—and adjust the contrastive loss accordingly. This prevents the model from being penalized for recognizing legitimate semantic similarities between different samples.

Marginal Sample Enhancement

We further enhance the model's performance using a triplet loss focused on "hard negatives." These are samples that are semantically close to the query but are not a match. This forces the model to learn more precise boundaries between similar but distinct concepts.

Experimental Results

HunYuan_tvr was evaluated across five major benchmarks, setting new state-of-the-art (SOTA) records. On the MSR-VTT benchmark, it achieved a Rank@1 of 55.0%, significantly outperforming previous methods like CLIP4Clip.

Dataset Rank@1 (%)
MSR-VTT 55.0
MSVD 57.8
LSMDC 29.7
DiDemo 52.1
ActivityNet 57.3

Scaling the model with a Sparsely Gated Mixture-of-Experts (SMoE) to 17 billion parameters further pushed the MSR-VTT Rank@1 to 62.9%, demonstrating the architecture's scalability.

Conclusion

HunYuan_tvr demonstrates that hierarchical alignment and noise-aware training are key to state-of-the-art text-video retrieval. By mimicking the human perception of actions as sequences of objects and events, the model achieves a more comprehensive multi-modal understanding. Future work will explore more sophisticated temporal aggregation mechanisms to further improve clip-phrase alignment.

Summary updated

Abstract

Text-Video Retrieval plays an important role in multi-modal understanding and has attracted increasing attention in recent years. Most existing methods focus on constructing contrastive pairs between whole videos and complete caption sentences, while ignoring fine-grained cross-modal relationships, e.g., short clips and phrases or single frame and word. In this paper, we propose a novel method, named HunYuan\_tvr, to explore hierarchical cross-modal interactions by simultaneously exploring video-sentence, clip-phrase, and frame-word relationships. Considering intrinsic semantic relations between frames, HunYuan\_tvr first performs self-attention to explore frame-wise correlations and adaptively clusters correlated frames into clip-level representations. Then, the clip-wise correlation is explored to aggregate clip representations into a compact one to describe the video globally. In this way, we can construct hierarchical video representations for frame-clip-video granularities, and also explore word-wise correlations to form word-phrase-sentence embeddings for the text modality. Finally, hierarchical contrastive learning is designed to explore cross-modal relationships,~\emph{i.e.,} frame-word, clip-phrase, and video-sentence, which enables HunYuan\_tvr to achieve a comprehensive multi-modal understanding. Further boosted by adaptive label denoising and marginal sample enhancement, HunYuan\_tvr obtains new state-of-the-art results on various benchmarks, e.g., Rank@1 of 55.0%, 57.8%, 29.7%, 52.1%, and 57.3% on MSR-VTT, MSVD, LSMDC, DiDemo, and ActivityNet respectively.
BibTeX
@inproceedings{min2025hunyuan,
  author = {Shaobo Min and Weijie Kong and Rong-Cheng Tu and Dihong Gong and Chengfei Cai and Wenzhe Zhao and Chenyang Liu and Sixiao Zheng and Hongfa Wang and Zhifeng Li and Wei Liu},
  title = {{HunYuan\_tvr for Text-Video Retrievial}},
  booktitle = {IEEE Access},
  year = {2025},
  url = {https://openreview.net/forum?id=ggm9f7HhhE},
}
Figure