Query-Based Video Summarization with Pseudo Label Supervision

Jia-Hong Huang, Luka Murn, Marta Mrak, Marcel Worring

Abstract

Query-based video summarization aims to generate concise video summaries tailored to specific user prompts. However, the field is often constrained by the scarcity of large-scale labeled datasets and the difficulty of aligning visual content with textual queries. This paper introduces a novel self-supervised approach that utilizes segment-level "pseudo labels" and a Transformer-based "semantics booster" to improve query understanding and modality interaction. The proposed method achieves state-of-the-art performance on major benchmarks, including TVSum and QueryVS.

Introduction

Video summarization is essential for distilling long-form content into meaningful highlights. In query-based summarization, the challenge is twofold: the model must not only identify important segments but also ensure those segments are relevant to a specific text prompt (e.g., "show me the goals" in a sports broadcast).

Currently, the field faces three significant hurdles:

  1. Data Scarcity: Manual labeling for video-query pairs is prohibitively expensive, leading to small, limited datasets.
  2. Ineffective Query Embedding: Traditional static word embeddings often fail to capture the nuanced contextual meaning of user prompts.
  3. Modality Interaction Gap: Models frequently struggle to effectively bridge the gap between visual pixels and textual semantics, making it difficult to pinpoint query-relevant segments.

Methodology

The authors propose a multi-pronged architecture designed to address these challenges through self-supervision and enhanced feature fusion.

Pseudo Label Supervision

To overcome data scarcity, the authors developed a pretext task using "pseudo labels." By taking existing frame-level human labels and averaging them over two-second intervals, they created segment-level supervision. This allows the model to learn broader temporal context and the general "rhythm" of video importance during a pre-training phase before it is fine-tuned on specific query-based tasks.

Semantics Booster

Rather than relying on static vectors, the model incorporates a Transformer-based "semantics booster." This module generates context-aware representations of the query, ensuring that the relationship between words in a prompt is fully captured, which is vital for handling complex or nuanced user requests.

Visual and Mutual Attention

The architecture utilizes a dual-stream visual processing approach: a 2D ResNet captures spatial details (objects and colors), while a 3D ResNet captures motion and temporal changes. These features are then integrated with the textual query via a Mutual Attention Mechanism. This convolutional attention layer acts as the "glue" between modalities, helping the model identify the specific visual segments that correspond to the text features.

Experimental Results

The researchers evaluated the model on three standard benchmarks: TVSum, SumMe, and QueryVS. The results indicate that the proposed method significantly outperforms previous state-of-the-art (SOTA) models.

  • TVSum: The model achieved an $F_1$-score of 68.4, a substantial increase over the previous best of 64.5 (SMN) and a massive jump from the baseline of 47.5.
  • QueryVS: The model reached an $F_1$-score of 55.3, surpassing the previous SOTA (GPT2MVS at 54.8).
  • Ablation Studies: These tests confirmed that every component contributed to the success. Notably, adding pseudo-label pre-training alone boosted the TVSum score from 47.5 to 61.3, demonstrating the power of weak supervision.

Discussion and Future Work

The success of this approach highlights that segment-level pseudo labels provide a powerful initialization for video models. Furthermore, the transition from static word vectors to context-aware Transformers proved critical for query relevance.

However, the authors acknowledge certain limitations. The pseudo labels, being automatically generated, can introduce noise or bias into the training process. Additionally, multi-modal feature fusion remains a complex bottleneck. Future research will likely focus on more sophisticated fusion techniques and the development of larger-scale video-query benchmarks to further push the boundaries of the field.

Conclusion

By combining self-supervised pseudo-labeling with advanced attention mechanisms and context-aware query processing, this research provides a robust framework for query-based video summarization. The significant performance gains on TVSum and QueryVS suggest that addressing the data gap through pretext tasks is a highly effective strategy for multi-modal video understanding.

Summary updated

BibTeX
@inproceedings{huang2023query,
  author = {Jia-Hong Huang and Luka Murn and Marta Mrak and Marcel Worring},
  title = {{Query-Based Video Summarization with Pseudo Label Supervision}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222138},
}
Figure