Video Summarization Based on Video-text Modelling

Haopeng Li, Qiuhong Ke, Mingming Gong, Rui ZhangarXiv 2022 · arXiv · OpenReview

Abstract

Modern video summarization methods are based on deep neural networks that require a large amount of annotated data for training. However, existing datasets for video summarization are small-scale, easily leading to over-fitting of the deep models. Considering that the annotation of large-scale datasets is time-consuming, we propose a multimodal self-supervised learning framework to obtain semantic representations of videos, which benefits the video summarization task. Specifically, the self-supervised learning is conducted by exploring the semantic consistency between the videos and text in both coarse-grained and fine-grained fashions, as well as recovering masked frames in the videos. The multimodal framework is trained on a newly-collected dataset that consists of video-text pairs. Additionally, we introduce a progressive video summarization method, where the important content in a video is pinpointed progressively to generate better summaries. Extensive experiments have proved the effectiveness and superiority of our method in rank correlation coefficients and F-score.

Introduction

Video summarization aims to condense long videos into concise versions that retain the most important semantic content. While deep learning has advanced this field, the primary bottleneck remains data scarcity. Standard benchmarks like SumMe and TVSum contain very few videos, making it difficult to train robust models without over-fitting. Manually annotating frame-level importance scores is subjective and labor-intensive.

To address this, we propose leveraging the abundance of unlabelled video-text pairs available online. By pre-training a model to understand the relationship between visual content and textual descriptions (like titles and queries), we can develop a semantic understanding of video importance that generalizes well to summarization tasks.

Methodology

Our approach consists of two main components: Multimodal Self-supervised Pre-training and Progressive Summarization.

Multimodal Self-supervised Learning (SSL)

We utilize a dual-encoder architecture (Text Encoder $E_T$ and Video Encoder $E_V$) to learn from three auxiliary tasks:

  1. Coarse-grained Modeling: Aligns global representations of video and text sequences.
  2. Fine-grained Modeling: Uses Hausdorff distance to measure similarity between individual words and video frames, forcing the model to capture local correlations.
  3. Temporal Dependency Capturing: A masked-frame recovery task where the model predicts missing frames based on local and global context, similar to masked language modeling in NLP.
The architecture of the proposed multimodal self-supervised learning framework, highlighting the coarse-grained, fine-grained, and temporal modeling tasks.

Progressive Video Summarization (SSPVS)

Traditional summarizers often struggle to pinpoint key content in a single pass. We introduce a multi-stage architecture where importance scores from one stage are used to weight the input features for the next. This iterative refinement allows the model to "zoom in" on relevant segments, correcting early errors and sharpening the final summary.

Experimental Setup

We collected a new dataset, YTVT (YouTube Video-Text), containing 3,081 video-text pairs for pre-training. This dataset includes rich metadata such as categories, search queries, titles, and descriptions. For evaluation, we use the standard SumMe and TVSum benchmarks, focusing on rank correlation coefficients (Spearman’s $ ho$ and Kendall’s $ au$) and F-scores.

Key Results

The proposed SSPVS model demonstrates significant improvements over state-of-the-art (SOTA) methods. As shown in the table below, our method achieves a substantial jump in rank correlation metrics. On SumMe, Spearman’s $ ho$ increased from a previous best of 0.085 to 0.240. When text metadata is included during inference (SSPVS+Text), performance reaches 0.257.

Comparison of rank correlation coefficients (Kendall’s $\tau$ and Spearman’s $\rho$) on SumMe and TVSum datasets against various baselines.

Ablation studies confirmed that each stage of the progressive refinement contributes to the final performance, with the optimal balance of accuracy and efficiency typically found at 3 or 4 stages. Furthermore, the model achieved a competitive F-score of 0.507 on SumMe, outperforming both supervised and unsupervised baselines.

Discussion and Conclusion

Our findings suggest that text serves as a powerful semantic anchor for video understanding. By aligning visual encoders with textual summaries during pre-training, the model naturally learns to prioritize semantically rich frames. The progressive refinement strategy further mimics human behavior—skimming a video before focusing on critical chapters.

While the multi-stage approach increases computational overhead, the gains in accuracy and ranking quality are substantial. Future work will explore more complex fusion techniques and the integration of larger pre-trained backbones to further enhance the semantic depth of automated video summaries.

Summary updated

Abstract

Modern video summarization methods are based on deep neural networks that require a large amount of annotated data for training. However, existing datasets for video summarization are small-scale, easily leading to over-fitting of the deep models. Considering that the annotation of large-scale datasets is time-consuming, we propose a multimodal self-supervised learning framework to obtain semantic representations of videos, which benefits the video summarization task. Specifically, the self-supervised learning is conducted by exploring the semantic consistency between the videos and text in both coarse-grained and fine-grained fashions, as well as recovering masked frames in the videos. The multimodal framework is trained on a newly-collected dataset that consists of video-text pairs. Additionally, we introduce a progressive video summarization method, where the important content in a video is pinpointed progressively to generate better summaries. Extensive experiments have proved the effectiveness and superiority of our method in rank correlation coefficients and F-score.
BibTeX
@misc{li2022video,
  author = {Haopeng Li and Qiuhong Ke and Mingming Gong and Rui Zhang},
  title = {{Video Summarization Based on Video-text Modelling}},
  year = {2022},
  eprint = {2201.02494},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.02494},
}
Figure