MS-DETR: Towards Effective Video Moment Retrieval and Highlight Detection by Joint Motion-Semantic Learning

Hongxu Ma, Guanshuo Wang, Fufu Yu, Qiong Jia, Shouhong Ding

Abstract

Video Moment Retrieval (MR) and Highlight Detection (HD) are pivotal for navigating large-scale video content. However, current approaches often fail to distinguish between spatial semantics and temporal motion, while also suffering from the "sparsity dilemma" where brief user queries lack the detail necessary for precise localization. We introduce MS-DETR, a framework that disentangles motion and semantics through a dual-tower encoder and fosters synergy between MR and HD via a collaborated decoder. Furthermore, we employ Large Language Models (LLMs) to generate dense auxiliary captions, mitigated by a contrastive denoising strategy. MS-DETR establishes new state-of-the-art benchmarks on QVHighlights, Charades-STA, TACoS, and TVSum.

Introduction

Video understanding requires identifying both "what" is in a scene (spatial semantics) and "how" it is changing (temporal motion). For instance, a query like "a man in a white t-shirt holding a fork and eating" contains semantic identifiers (man, white t-shirt) and motion-based actions (holding a fork, eating). Most existing models aggregate these into a single global feature, leading to sub-optimal alignment. Additionally, the limited descriptive power of human-provided queries—the "sparsity dilemma"—hinders the model's ability to learn fine-grained nuances.

Proposed Method: MS-DETR

The MS-DETR architecture addresses these challenges through three primary innovations:

  1. Motion-Semantics Disentangled Encoder (MSDE): Instead of a monolithic transformer, MSDE utilizes two specialized branches. One branch extracts temporal motion features (e.g., via SlowFast), while the other focuses on spatial semantics (e.g., via CLIP). These features interact independently with the text query before fusion, ensuring that motion-heavy or semantic-heavy queries are processed with appropriate focus.
  2. Mutual Task-Collaborated Decoder (MTCD): MR and HD are inherently related. MTCD exploits this by using HD salience scores as priors for the MR task and using MR boundary predictions to refine the salience scores for HD. This mutual feedback loop improves the precision of both tasks.
  3. Contrastive Denoising Learning (CDL): To overcome data sparsity, the authors used LLaVA to generate detailed auxiliary captions and hard-negative queries. To manage potential LLM hallucinations, a Contrastive Denoising strategy is applied, ensuring the model learns from high-quality, robust features.
The MS-DETR architecture featuring the Motion-Semantics Disentangled Encoder (MSDE) and Mutual Task-Collaborated Decoder (MTCD).

Experimental Results

MS-DETR was evaluated on several major benchmarks, including QVHighlights, Charades-STA, TACoS, and TVSum. On the QVHighlights dataset, MS-DETR achieved a significant performance boost, reaching a 44.89 average mAP on the test set, surpassing previous state-of-the-art models like CG-DETR (42.86).

Ablation studies confirm that each component—MSDE, MTCD, and the LLM-based data augmentation—contributes significantly to the final performance. Notably, the combination of all modules yields the best results across both Moment Retrieval (MR) and Highlight Detection (HD) metrics.

Ablation study demonstrating the contribution of each component (MSDE, MTCD, AD, CDL) to the overall performance on the QVHighlights dataset.

Conclusion

MS-DETR demonstrates that explicitly disentangling motion and semantics, combined with task-level collaboration and LLM-driven data augmentation, significantly advances video moment retrieval and highlight detection. While currently limited to visual and textual modalities, future iterations aim to integrate audio features to further enhance the system's robustness in complex multi-modal environments.

Summary updated

BibTeX
@inproceedings{ma2025ms,
  author = {Hongxu Ma and Guanshuo Wang and Fufu Yu and Qiong Jia and Shouhong Ding},
  title = {{MS-DETR: Towards Effective Video Moment Retrieval and Highlight Detection by Joint Motion-Semantic Learning}},
  booktitle = {ACM Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=l7P1BsAGbp},
}
Figure