Temporal Cue Guided Video Highlight Detection with Low-Rank Audio-Visual Fusion

Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, Guang Yang

Abstract

Identifying highlight moments in long, untrimmed videos is a critical task for content creators. This paper introduces a weakly supervised framework that learns to detect highlights using only video-level topic tags. The proposed model integrates a low-rank audio-visual tensor fusion mechanism with a hierarchical temporal context encoder to capture complex multi-modal interactions and essential "before-and-after" temporal cues. Evaluated on major benchmarks, the framework achieves state-of-the-art results, demonstrating significant improvements in efficiency and accuracy.

Introduction

The explosion of short-form video content on platforms like TikTok and Reels has intensified the need for automated highlight detection. Traditional supervised methods require expensive frame-level annotations. Weakly supervised approaches offer a scalable alternative by training on video-level tags (e.g., "surfing"), but they often struggle with three main challenges:

  1. Multi-modal Complexity: Fusing audio and visual signals efficiently is difficult.
  2. Temporal Blindness: Ignoring the context surrounding a highlight makes it hard to distinguish from mundane moments.
  3. The MIL Bottleneck: Standard Multiple Instance Learning (MIL) often suffers from gradient vanishing, where a single high-scoring segment prevents the model from learning from other relevant parts.
Temporal context helps distinguish highlights (B) from non-highlights (A) by encoding the 'start' and 'end' cues surrounding an event.

As shown in Figure 1, context is vital. A surfer "wiping out" or "taking off" provides the temporal cues necessary to identify a "surfing highlight." Modeling this temporal evolution is more effective than looking for "exciting" pixels in a vacuum.

Proposed Methodology

The architecture consists of three primary components designed to address the aforementioned hurdles.

The complete framework architecture, showing the progression from raw video to audio-visual fusion, hierarchical encoding, and final instance aggregation.

1. Low-Rank Audio-Visual Tensor Fusion

Rather than simple concatenation or memory-intensive bilinear pooling, the model employs tensor decomposition with a low-rank constraint. This captures high-level interactions between audio and visual features while reducing the number of trainable parameters by approximately 85% compared to compact bilinear pooling.

2. Hierarchical Temporal Context Encoder

This module aggregates information from neighboring segments to "see" the preparation and aftermath of an event. By modeling the temporal evolution of a video, the system can better distinguish true highlights from isolated exciting-looking frames. Ablation studies showed that a neighbor size of $k=3$ was the "sweet spot" for performance.

3. Attention-Gated Instance Aggregation

To overcome the gradient vanishing problem in weak supervision, the authors replaced standard "Noisy-OR" logic with an attention-gated module. This ensures that gradients flow back to multiple segments during training, allowing the model to learn from all relevant parts of the video rather than getting "stuck" on a single snippet.

Experimental Results

The model was evaluated on two gold-standard benchmarks:

  • YouTube Highlights: The method achieved a Mean Average Precision (mAP) of 62.97%, a 4.6% improvement over the previous state-of-the-art multi-modal method, MINI-Net.
  • TVSum: The model reached a Top-5 mAP of 76.82%, representing a 6.62% relative gain over MINI-Net.

Conclusion and Future Work

This research demonstrates that modeling temporal context and using efficient low-rank fusion can significantly advance weakly supervised highlight detection. While the model currently relies on pre-defined topic tags, future work aims to develop topic-agnostic detectors and explore multi-scale segment lengths to capture highlights of varying durations more effectively.

Summary updated

Abstract

Video highlight detection plays an increasingly important role in social media content filtering, however, it remains highly challenging to develop automated video highlight detection methods because of the lack of temporal annotations (i.e., where the highlight moments are in long videos) for supervised learning. In this paper, we propose a novel weakly supervised method that can learn to detect highlights by mining video characteristics with video level annotations (topic tags) only. Particularly, we exploit audio-visual features to enhance video representation and take temporal cues into account for improving detection performance. Our contributions are threefold: 1) we propose an audio-visual tensor fusion mechanism that efficiently models the complex association between two modalities while reducing the gap of the heterogeneity between the two modalities; 2) we introduce a novel hierarchical temporal context encoder to embed local temporal clues in between neighboring segments; 3) finally, we alleviate the gradient vanishing problem theoretically during model optimization with attention-gated instance aggregation. Extensive experiments on two benchmark datasets (YouTube Highlights and TVSum) have demonstrated our method outperforms other state-of-the-art methods with remarkable improvements.
BibTeX
@inproceedings{ye2021temporal,
  author = {Qinghao Ye and Xiyue Shen and Yuan Gao and Zirui Wang and Qi Bi and Ping Li and Guang Yang},
  title = {{Temporal Cue Guided Video Highlight Detection with Low-Rank Audio-Visual Fusion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00785},
}
Figure