Unsupervised Video Summarization with Attentive Conditional Generative Adversarial Networks

Xufeng He, Yang Hua, Tao Song, Zongpu Zhang, Zhengui Xue, Ruhui Ma, Neil Martin Robertson, Haibing Guan

Abstract

This paper introduces the Attentive Conditional Generative Adversarial Network (ACGAN), a novel unsupervised framework for video summarization. ACGAN addresses the limitations of existing reconstruction-based GAN models, such as information loss and local temporal bias, by integrating multi-head self-attention (MHSA) and a conditional feature selector. Experimental results on the SumMe and TVSum datasets demonstrate that ACGAN achieves state-of-the-art performance in the unsupervised category and remains highly competitive against supervised methods, effectively identifying key frames without the need for human annotations.

Introduction

The exponential growth of video content on digital platforms has created an urgent need for automated summarization tools. Traditional approaches often frame this as a supervised learning task, requiring extensive human-curated summaries. However, human labeling is not only resource-intensive but also inherently subjective, as different annotators rarely agree on a single "perfect" summary.

Unsupervised methods offer a scalable alternative, but current Generative Adversarial Network (GAN) architectures face significant challenges. Many rely on an encoder-decoder structure to "reconstruct" the original video from a summary, a process that is mathematically prone to information loss. Furthermore, these models often depend on Recurrent Neural Networks (like LSTMs), which struggle to capture long-range dependencies in long videos, leading to a local temporal bias.

Methodology

The proposed ACGAN architecture moves away from pixel-level reconstruction, focusing instead on feature matching within an adversarial framework.

Architecture Overview

The ACGAN consists of two primary components:

  1. Generator: Utilizing a combination of Bidirectional LSTMs (BiLSTMs) and Multi-head Self-Attention (MHSA), the generator predicts importance scores for each frame based on features extracted from a pre-trained CNN. It produces "weighted frame features" that represent the summary.
  2. Discriminator: The discriminator is trained to distinguish between the original raw frame features and the weighted features generated by the model.
Architecture of the ACGAN framework

Key Innovations

  • Multi-head Self-Attention (MHSA): By incorporating MHSA, the model can attend to all positions in a sequence simultaneously. This allows the model to capture global context across the entire video, overcoming the "forgetting" problem associated with standard LSTMs.
  • Conditional Feature Selector (CF): This module uses attention scores to select a subset of high-impact features. These are then fed back into the GAN as conditional information, guiding the adversarial learning process to focus on the most significant temporal regions.

Experimental Results

The model was evaluated on the SumMe and TVSum benchmarks, which include diverse video categories ranging from sports to cooking. ACGAN achieved an F-score of 46.0% on SumMe and 58.5% on TVSum, setting new benchmarks for unsupervised summarization.

Comparison of ACGAN with state-of-the-art unsupervised methods

Beyond unsupervised benchmarks, ACGAN proved remarkably competitive with supervised models. For instance, its 46.0% score on SumMe exceeds the performance of supervised Bi-LSTM (37.6%) and DPP-LSTM (38.6%) models. Transferability tests—training on one dataset and testing on another—further confirmed that the model learns generalizable concepts of visual importance rather than dataset-specific patterns.

Conclusion

ACGAN represents a significant advancement in unsupervised video summarization. By combining global attention mechanisms with conditional adversarial learning, the model effectively identifies key frames without the need for human labels. While the model shows some sensitivity to variable video lengths during training, its ability to outperform supervised benchmarks highlights the potential of expressive unsupervised architectures. Future research will explore stabilizing training for extreme temporal variations and applying more advanced feature descriptors to further enhance summarization quality.

Summary updated

Abstract

With the rapid growth of video data, video summarization technique plays a key role in reducing people's efforts to explore the content of videos by generating concise but informative summaries. Though supervised video summarization approaches have been well studied and achieved state-of-the-art performance, unsupervised methods are still highly demanded due to the intrinsic difficulty of obtaining high-quality annotations. In this paper, we propose a novel yet simple unsupervised video summarization method with attentive conditional Generative Adversarial Networks (GANs). Firstly, we build our framework upon Generative Adversarial Networks in an unsupervised manner. Specifically, the generator produces high-level weighted frame features and predicts frame-level importance scores, while the discriminator tries to distinguish between weighted frame features and raw frame features. Furthermore, we utilize a conditional feature selector to guide GAN model to focus on more important temporal regions of the whole video frames. Secondly, we are the first to introduce the frame-level multi-head self-attention for video summarization, which learns long-range temporal dependencies along the whole video sequence and overcomes the local constraints of recurrent units, e.g., LSTMs. Extensive evaluations on two datasets, SumMe and TVSum, show that our proposed framework surpasses state-of-the-art unsupervised methods by a large margin, and even outperforms most of the supervised methods. Additionally, we also conduct the ablation study to unveil the influence of each component and parameter settings in our framework.
BibTeX
@inproceedings{he2019unsupervised,
  author = {Xufeng He and Yang Hua and Tao Song and Zongpu Zhang and Zhengui Xue and Ruhui Ma and Neil Martin Robertson and Haibing Guan},
  title = {{Unsupervised Video Summarization with Attentive Conditional Generative Adversarial Networks}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3351056},
}
Figure