Abstract
Text-Video Retrieval plays an important role in multi-modal understanding and has attracted increasing attention in recent years. Most existing methods focus on constructing contrastive pairs between whole videos and complete caption sentences, while ignoring fine-grained cross-modal relationships, e.g., short clips and phrases or single frame and word. In this paper, we propose a novel method, named HunYuan_tvr, to explore hierarchical cross-modal interactions by simultaneously exploring video-sentence, clip-phrase, and frame-word relationships. Considering intrinsic semantic relations between frames, HunYuan_tvr first performs self-attention to explore frame-wise correlations and adaptively clusters correlated frames into clip-level representations. Then, the clip-wise correlation is explored to aggregate clip representations into a compact one to describe the video globally. In this way, we can construct hierarchical video representations for frame-clip-video granularities, and also explore word-wise correlations to form word-phrase-sentence embeddings for the text modality. Finally, hierarchical contrastive learning is designed to explore cross-modal relationships, i.e., frame-word, clip-phrase, and video-sentence, which enables HunYuan_tvr to achieve a comprehensive multi-modal understanding. Further boosted by adaptive label denoising and marginal sample enhancement, HunYuan_tvr obtains new state-of-the-art results on various benchmarks, e.g., Rank@1 of 55.0%, 57.8%, 29.7%, 52.1%, and 57.3% on MSR-VTT, MSVD, LSMDC, DiDemo, and ActivityNet respectively.
Introduction
Text-Video Retrieval (TVR) is a fundamental task in multi-modal understanding, aiming to retrieve relevant videos for a given text query. While models like CLIP have significantly advanced image-text alignment, extending this to video introduces the challenge of temporal dynamics. Current TVR approaches often rely on global matching, which overlooks fine-grained details, or simple token-level matching, which misses high-level semantic context.
Furthermore, standard contrastive learning frameworks often suffer from the "noisy label" problem. In a typical training batch, all samples except the ground truth are treated as negatives. However, multiple videos might share similar semantic content (e.g., "a person running"), leading to false negatives that hinder the model's ability to learn nuanced features.
Hierarchical Cross-Modal Interaction
To address these limitations, we introduce HunYuan_tvr, a framework centered on Hierarchical Cross-Modal Interaction (HCMI). Instead of a single global vector, HCMI constructs a hierarchy for both visual and textual modalities.
Multi-Level Representations
The model processes data at three distinct granularities:
- Frame-Word Level: Captures fine-grained object-level interactions.
- Clip-Phrase Level: Aggregates frames into clips via self-attention to capture short-term actions and matches them with textual phrases.
- Video-Sentence Level: Provides a global summary of the entire video and sentence.
The visual hierarchy is built by adaptively clustering correlated frames into clip-level representations, which are then further aggregated into a global video embedding. A similar process is applied to the text modality to form word-phrase-sentence embeddings.
Training Strategy and Optimization
Beyond the hierarchical architecture, HunYuan_tvr employs advanced training strategies to improve robustness and discriminative power.
Adaptive Label Denoising
To mitigate the false negative problem, we utilize an adaptive label denoising strategy. By comparing different random samplings of the same video against other videos in the batch, the model can identify "potential positives"—videos that are semantically similar to the target—and adjust the contrastive loss accordingly. This prevents the model from being penalized for recognizing legitimate semantic similarities between different samples.
Marginal Sample Enhancement
We further enhance the model's performance using a triplet loss focused on "hard negatives." These are samples that are semantically close to the query but are not a match. This forces the model to learn more precise boundaries between similar but distinct concepts.
Experimental Results
HunYuan_tvr was evaluated across five major benchmarks, setting new state-of-the-art (SOTA) records. On the MSR-VTT benchmark, it achieved a Rank@1 of 55.0%, significantly outperforming previous methods like CLIP4Clip.
| Dataset | Rank@1 (%) |
|---|---|
| MSR-VTT | 55.0 |
| MSVD | 57.8 |
| LSMDC | 29.7 |
| DiDemo | 52.1 |
| ActivityNet | 57.3 |
Scaling the model with a Sparsely Gated Mixture-of-Experts (SMoE) to 17 billion parameters further pushed the MSR-VTT Rank@1 to 62.9%, demonstrating the architecture's scalability.
Conclusion
HunYuan_tvr demonstrates that hierarchical alignment and noise-aware training are key to state-of-the-art text-video retrieval. By mimicking the human perception of actions as sequences of objects and events, the model achieves a more comprehensive multi-modal understanding. Future work will explore more sophisticated temporal aggregation mechanisms to further improve clip-phrase alignment.
Summary updated