Abstract
Video Moment Retrieval (MR) and Highlight Detection (HD) are pivotal for navigating large-scale video content. However, current approaches often fail to distinguish between spatial semantics and temporal motion, while also suffering from the "sparsity dilemma" where brief user queries lack the detail necessary for precise localization. We introduce MS-DETR, a framework that disentangles motion and semantics through a dual-tower encoder and fosters synergy between MR and HD via a collaborated decoder. Furthermore, we employ Large Language Models (LLMs) to generate dense auxiliary captions, mitigated by a contrastive denoising strategy. MS-DETR establishes new state-of-the-art benchmarks on QVHighlights, Charades-STA, TACoS, and TVSum.
Introduction
Video understanding requires identifying both "what" is in a scene (spatial semantics) and "how" it is changing (temporal motion). For instance, a query like "a man in a white t-shirt holding a fork and eating" contains semantic identifiers (man, white t-shirt) and motion-based actions (holding a fork, eating). Most existing models aggregate these into a single global feature, leading to sub-optimal alignment. Additionally, the limited descriptive power of human-provided queries—the "sparsity dilemma"—hinders the model's ability to learn fine-grained nuances.
Proposed Method: MS-DETR
The MS-DETR architecture addresses these challenges through three primary innovations:
- Motion-Semantics Disentangled Encoder (MSDE): Instead of a monolithic transformer, MSDE utilizes two specialized branches. One branch extracts temporal motion features (e.g., via SlowFast), while the other focuses on spatial semantics (e.g., via CLIP). These features interact independently with the text query before fusion, ensuring that motion-heavy or semantic-heavy queries are processed with appropriate focus.
- Mutual Task-Collaborated Decoder (MTCD): MR and HD are inherently related. MTCD exploits this by using HD salience scores as priors for the MR task and using MR boundary predictions to refine the salience scores for HD. This mutual feedback loop improves the precision of both tasks.
- Contrastive Denoising Learning (CDL): To overcome data sparsity, the authors used LLaVA to generate detailed auxiliary captions and hard-negative queries. To manage potential LLM hallucinations, a Contrastive Denoising strategy is applied, ensuring the model learns from high-quality, robust features.
Experimental Results
MS-DETR was evaluated on several major benchmarks, including QVHighlights, Charades-STA, TACoS, and TVSum. On the QVHighlights dataset, MS-DETR achieved a significant performance boost, reaching a 44.89 average mAP on the test set, surpassing previous state-of-the-art models like CG-DETR (42.86).
Ablation studies confirm that each component—MSDE, MTCD, and the LLM-based data augmentation—contributes significantly to the final performance. Notably, the combination of all modules yields the best results across both Moment Retrieval (MR) and Highlight Detection (HD) metrics.
Conclusion
MS-DETR demonstrates that explicitly disentangling motion and semantics, combined with task-level collaboration and LLM-driven data augmentation, significantly advances video moment retrieval and highlight detection. While currently limited to visual and textual modalities, future iterations aim to integrate audio features to further enhance the system's robustness in complex multi-modal environments.
Summary updated