Introduction
In the field of video understanding, two tasks are fundamentally linked yet distinct in their objectives: Moment Retrieval (MR) and Highlight Detection (HD). Moment Retrieval involves localizing the specific start and end times of a video segment that corresponds to a natural language query. In contrast, Highlight Detection focuses on identifying the most salient or "exciting" parts of a video relative to that query.
Most existing models adopt a "bottom-up" approach, using a shared backbone to extract common features for both tasks. However, this often overlooks the unique requirements of each task and fails to leverage the potential synergy between them. Researchers from Xi’an Jiaotong University have introduced TaskWeave, a novel framework that decouples task-specific representations and enables a bidirectional feedback loop, allowing the two tasks to inform and refine each other.
The TaskWeave Framework
TaskWeave moves away from simple feature fusion toward a "top-down" architecture designed to handle task specificity and inter-task interplay through three core innovations.
Task-Decoupled Unit
Instead of relying on a single shared feature pool, TaskWeave employs a multi-expert strategy. A shared expert captures common multi-modal features, while MR-specific and HD-specific experts focus on the unique requirements of temporal localization and saliency detection, respectively. These specialized features are combined via element-wise products to create task-aware representations ($X_{mr}$ and $X_{hd}$).
Inter-Task Feedback Mechanism
The "weaving" in TaskWeave refers to its bidirectional feedback loop:
- Moment-to-Mask (M2M): Predicted moments from the MR head are converted into temporal masks. These masks guide the HD head to focus on relevant intervals, ensuring highlight detection is grounded within the retrieved moment.
- Highlightness-to-Mask (H2M): Saliency scores from the HD head are binarized into masks that help the MR head refine its temporal boundaries. High saliency scores act as strong priors for the presence of the target moment.
Task-Dependent Joint Loss
To prevent a "seesaw effect"—where improving performance in one task degrades the other—TaskWeave utilizes a dynamic weighting scheme for the joint loss. Rather than manual tuning, the model adjusts task weights based on the uncertainty of each task during training, using a principled approach based on Gaussian and Boltzmann distributions.
Experimental Evaluation
The framework was evaluated on three major benchmarks: QVHighlights, Charades-STA, and TVSum, setting new state-of-the-art (SOTA) records across the board.
Key Results
- QVHighlights: TaskWeave achieved a 64.26% Recall@1 (0.5 IoU) and a significant 8.72% improvement in Average mAP for Moment Retrieval compared to previous models like QD-DETR and UMT.
- TVSum: The model outperformed existing methods in 9 out of 10 categories, showing a 2.71% improvement in Average mAP for highlight detection.
- Charades-STA: Using a VGG backbone, the model reached 56.51% [email protected], a substantial gain over prior joint-task architectures.
Qualitative analysis indicates that TaskWeave is particularly effective at handling "disjoint moments," where a single query corresponds to multiple non-contiguous segments in a video.
Conclusion
TaskWeave demonstrates that decoupling task-specific features and enabling bidirectional feedback significantly enhances joint moment retrieval and highlight detection. By treating the output of one task as a prior for the other, the model achieves a more holistic understanding of video content. While the current version focuses on visual and text modalities, future work may incorporate audio features and optimize feedback timing to further improve robustness and inference speed.
Summary updated