Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection

Jin Yang, Ping Wei, Huan Li, Ziyang RenCVPR 2024

Introduction

In the field of video understanding, two tasks are fundamentally linked yet distinct in their objectives: Moment Retrieval (MR) and Highlight Detection (HD). Moment Retrieval involves localizing the specific start and end times of a video segment that corresponds to a natural language query. In contrast, Highlight Detection focuses on identifying the most salient or "exciting" parts of a video relative to that query.

Visualizing the relationship between Moment Retrieval and Highlight Detection.

Most existing models adopt a "bottom-up" approach, using a shared backbone to extract common features for both tasks. However, this often overlooks the unique requirements of each task and fails to leverage the potential synergy between them. Researchers from Xi’an Jiaotong University have introduced TaskWeave, a novel framework that decouples task-specific representations and enables a bidirectional feedback loop, allowing the two tasks to inform and refine each other.

The TaskWeave Framework

TaskWeave moves away from simple feature fusion toward a "top-down" architecture designed to handle task specificity and inter-task interplay through three core innovations.

The TaskWeave architecture, illustrating the Task-Decoupled Unit and the Inter-task Feedback Mechanism.

Task-Decoupled Unit

Instead of relying on a single shared feature pool, TaskWeave employs a multi-expert strategy. A shared expert captures common multi-modal features, while MR-specific and HD-specific experts focus on the unique requirements of temporal localization and saliency detection, respectively. These specialized features are combined via element-wise products to create task-aware representations ($X_{mr}$ and $X_{hd}$).

Inter-Task Feedback Mechanism

The "weaving" in TaskWeave refers to its bidirectional feedback loop:

  • Moment-to-Mask (M2M): Predicted moments from the MR head are converted into temporal masks. These masks guide the HD head to focus on relevant intervals, ensuring highlight detection is grounded within the retrieved moment.
  • Highlightness-to-Mask (H2M): Saliency scores from the HD head are binarized into masks that help the MR head refine its temporal boundaries. High saliency scores act as strong priors for the presence of the target moment.

Task-Dependent Joint Loss

To prevent a "seesaw effect"—where improving performance in one task degrades the other—TaskWeave utilizes a dynamic weighting scheme for the joint loss. Rather than manual tuning, the model adjusts task weights based on the uncertainty of each task during training, using a principled approach based on Gaussian and Boltzmann distributions.

Experimental Evaluation

The framework was evaluated on three major benchmarks: QVHighlights, Charades-STA, and TVSum, setting new state-of-the-art (SOTA) records across the board.

Key Results

  • QVHighlights: TaskWeave achieved a 64.26% Recall@1 (0.5 IoU) and a significant 8.72% improvement in Average mAP for Moment Retrieval compared to previous models like QD-DETR and UMT.
  • TVSum: The model outperformed existing methods in 9 out of 10 categories, showing a 2.71% improvement in Average mAP for highlight detection.
  • Charades-STA: Using a VGG backbone, the model reached 56.51% [email protected], a substantial gain over prior joint-task architectures.

Qualitative analysis indicates that TaskWeave is particularly effective at handling "disjoint moments," where a single query corresponds to multiple non-contiguous segments in a video.

Conclusion

TaskWeave demonstrates that decoupling task-specific features and enabling bidirectional feedback significantly enhances joint moment retrieval and highlight detection. By treating the output of one task as a prior for the other, the model achieves a more holistic understanding of video content. While the current version focuses on visual and text modalities, future work may incorporate audio features and optimize feedback timing to further improve robustness and inference speed.

Summary updated

BibTeX
@inproceedings{yang2024task,
  author = {Jin Yang and Ping Wei and Huan Li and Ziyang Ren},
  title = {{Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure