TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection

Hao Sun, Mingyao Zhou, Wenjing Chen, Wei Xie

Abstract

TR-DETR is a novel transformer-based framework designed to jointly address video Moment Retrieval (MR) and Highlight Detection (HD). By explicitly modeling the "reciprocal" relationship between these tasks, the model allows highlight scores to guide moment localization and retrieved moments to refine highlight distributions. Combined with early multi-modal alignment and query-guided feature refinement, TR-DETR achieves state-of-the-art performance on major benchmarks including QVHighlights, Charades-STA, and TVSum.

Introduction

In video understanding, researchers typically focus on two distinct but related tasks: Moment Retrieval (MR), which involves finding specific video segments based on a natural language query, and Highlight Detection (HD), which identifies the most relevant or "exciting" parts of a video relative to that query.

Historically, these tasks have been treated as independent outputs from a shared backbone, often using separate prediction heads at the end of a network. However, this approach overlooks a fundamental intuition: a high-quality highlight is inherently part of a retrieved moment, and the boundaries of a moment can help define what constitutes a highlight. TR-DETR (Task-Reciprocal Transformer) is designed to exploit this task reciprocity, allowing the two tasks to inform and improve each other.

Methodology

The TR-DETR architecture is built upon four primary stages designed to maximize the interaction between textual queries and visual content while leveraging the synergy between MR and HD tasks.

The TR-DETR architecture, highlighting the Local-Global Multi-Modal Alignment and the Task Cooperation Module (HD2MR and MR2HD).

1. Feature Extraction

The framework utilizes pre-trained, "frozen" networks to extract multi-modal features. Visual features are obtained via CLIP and SlowFast, while textual features come from CLIP's text encoder. Audio features, when used, are extracted via PANN.

2. Local-Global Multi-Modal Alignment

A key innovation is the early alignment of modalities. Before the transformer layers, a module uses contrastive learning with local and global regulators to map video clips and text words into a shared latent space. This "pre-alignment" helps the model distinguish between semantically similar but irrelevant clips, providing a cleaner signal for subsequent layers.

3. Visual Feature Refinement

To mitigate the impact of irrelevant background content, TR-DETR uses the text query to filter visual features. This query-guided filtering retains only the most relevant temporal cues, preventing the model from over-attending to visually "loud" but query-irrelevant segments.

4. Task Cooperation Module

The core of the framework is the Task Cooperation Module, which implements reciprocity in two directions:

  • HD2MR (Highlight Detection to Moment Retrieval): Predicted highlight scores are used to weight joint features, effectively filtering out non-highlight information to assist the retrieval head in finding precise moment boundaries.
  • MR2HD (Moment Retrieval to Highlight Detection): Global features from retrieved moments are used to refine the initial highlight distribution, ensuring that highlight scores remain consistent with the broader context of the retrieved content.

Experiments and Results

TR-DETR was evaluated on three major benchmarks: QVHighlights, Charades-STA, and TVSum. The model consistently outperformed existing state-of-the-art (SOTA) methods like QD-DETR and UniVTG.

Quantitative Performance

On the QVHighlights test set, TR-DETR demonstrated significant improvements, increasing [email protected] (Recall at a strict 0.7 IoU) by 3.98% and [email protected] by 3.75% over previous bests. In the Charades-STA dataset, it achieved a SOTA [email protected] of 55.51%. For TVSum, the model reached an average top-5 mAP of 88.1%, showcasing its ability to identify highlights even without extensive multi-modal pre-training.

Qualitative Analysis

Visualizations of the model's output reveal that TR-DETR produces smoother and more accurate saliency curves that align closely with human ground-truth annotations.

Qualitative comparison on the QVHighlights dataset. TR-DETR produces more accurate moment boundaries and saliency curves compared to the QD-DETR baseline.

As shown in the figure above, TR-DETR's predictions for both the moment interval and the saliency curve are more precise than those of the baseline QD-DETR, particularly in complex scenarios where the query requires fine-grained temporal understanding.

Conclusion

TR-DETR demonstrates that explicitly modeling task reciprocity between Moment Retrieval and Highlight Detection leads to superior performance in video understanding. By aligning modalities early and allowing tasks to cooperate, the model achieves new state-of-the-art results across multiple benchmarks.

One identified limitation is the current method of audio feature integration; simple concatenation sometimes led to performance degradation. Future research will focus on more sophisticated multi-modal interaction networks to better fuse video, text, and audio information without misalignment penalties.

Summary updated

BibTeX
@inproceedings{sun2024tr,
  author = {Hao Sun and Mingyao Zhou and Wenjing Chen and Wei Xie},
  title = {{TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28304},
}
Figure