Self-Supervised Learning to Detect Key Frames in Videos

Xiang Yan, Syed Zulqarnain Gilani, Mingtao Feng, Liang Zhang, Hanlin Qin, Ajmal Mian2020 · DOI

TL;DR

This paper introduces a novel self-supervised learning framework for automatically detecting key frames in videos, crucial for tasks like video summarization and action recognition. By leveraging a two-stream convolutional neural network and a unique automatic annotation strategy, the method bypasses the need for expensive manual labeling, achieving competitive results on standard action recognition and video summarization datasets.

What Problem Does This Paper Address?

Extracting "key frames" from videos is essential for many computer vision applications. These key frames are representative frames that capture the most important information in a video, reducing redundancy and computational load. For instance, in video summarization, key frames form a concise summary of the entire video. In action recognition, key frames can effectively represent the core actions.

Traditional methods for detecting key frames often rely on supervised learning, which requires large datasets of videos with manually annotated key frames. This manual annotation process is not only time-consuming and expensive but also prone to subjective errors and inconsistencies among annotators. The paper aims to overcome these limitations by proposing a self-supervised approach that can automatically learn to identify key frames without requiring human-labeled data.

Methods

The proposed method consists of two main parts: automatic key frame annotation and a self-supervised deep learning model for key frame detection.

  1. Automatic Key Frame Annotation: This part uses Linear Discriminant Analysis (LDA) to generate "ground truth" labels for training.
  2. Feature Extraction: For each frame in a video, both appearance features (from RGB frames) and motion features (from optical flow images) are extracted using a pre-trained VGG-16 network. These features are then concatenated to form a spatio-temporal feature vector.
  3. LDA Projection: LDA is applied to these fused features. The goal is to find a projection that minimizes the distance between frames within the same action class (intra-class variance) and maximizes the distance between frames from different action classes (inter-class variance).
  4. Uniqueness Score: After projection, a "uniqueness score" is calculated for each frame. This score represents how different a frame is from the mean of its class. Frames with higher uniqueness scores are considered more likely to be key frames.
  5. Self-Supervised Deep Learning Model for Key Frame Detection:
    • Two-Stream ConvNet: A two-stream convolutional neural network (CNN) is employed. One stream processes RGB frames (appearance), and the other processes optical flow images (motion). Both streams share a similar architecture, based on VGG-16, but with modified final layers.
  6. Feature Fusion: The features from the appearance and motion streams are concatenated.
  7. Regression Network: The concatenated features are fed into fully connected layers, culminating in a regression layer. This layer is trained to predict the uniqueness score (generated in the annotation step) for each frame.
  8. Key Frame Detection: Once trained, the network takes a video as input and outputs a score for each frame. A smooth spline fitting function is applied to these scores, and the peaks in the fitted curve are identified as key frames.

The self-supervised nature comes from using the automatically generated uniqueness scores as labels, eliminating the need for manual annotation.

Datasets

  • UCF101: This popular human action recognition dataset was used for training the automatic key frame annotation model and for evaluating the key frame detection performance. The authors randomly selected 200 videos for training and 300 videos for testing from this dataset.
  • VSUMM: This dataset, commonly used for video summarization, was used to further evaluate the generalizability of the proposed key frame detection method. The authors used 20% of the videos for testing, following established protocols.

Key Insights and Ideas

  • Bridging the Annotation Gap: The core innovation is the automatic annotation strategy using LDA. This effectively creates "pseudo ground truth" labels, enabling self-supervised learning without manual annotation, which is a significant bottleneck in this field.
  • Spatio-Temporal Feature Fusion: The paper emphasizes the importance of combining appearance and motion features. By fusing features from both RGB frames and optical flow, the model gains a more comprehensive understanding of video content and dynamics.
  • Regression for Uniqueness: Framing key frame detection as a regression problem (predicting a continuous uniqueness score) rather than classification allows for a more nuanced representation of frame importance.
  • Generalizability: The study demonstrates that a model trained on action videos (UCF101) can perform well on video summarization tasks (VSUMM), suggesting the learned features are broadly applicable.

Key Results

  • UCF101 Performance: The method achieved low error rates for both the number of detected key frames and their temporal location, with an average location error of less than one frame. Qualitative results showed that the detected key frames align well with the automatically generated ground truth and effectively represent the actions.
  • VSUMM Performance: On the VSUMM dataset, the proposed method achieved an F-score of 72.1%, outperforming several traditional and deep learning baselines, including SUM-GAN (by 9.6%) and a deep supervised approach (by 2.4%).
  • Qualitative Analysis: Visualizations showed that the detected key frames often correspond to significant moments in actions, such as lifting a leg, striking a ball, or taking off in a jump, and can decompose complex actions into simpler sub-events.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the detected key frames are similar to the ground truth, the method might detect slightly more key frames than the automatically annotated ones in some cases.
  • The automatic annotation relies on LDA, which assumes linear separability of classes.
  • The method was primarily trained on action videos and then tested on summarization videos, suggesting potential for further domain-specific fine-tuning.
  • Future Work:
    • The authors suggest that the learned features can be used for other video understanding tasks.
  • The paper's success in demonstrating generalizability hints at exploring more diverse datasets and tasks.
  • Further research could investigate more sophisticated methods for automatic annotation or explore different network architectures for improved performance.

Summary updated

BibTeX
@misc{yan2020self,
  author = {Xiang Yan and Syed Zulqarnain Gilani and Mingtao Feng and Liang Zhang and Hanlin Qin and Ajmal Mian},
  title = {{Self-Supervised Learning to Detect Key Frames in Videos}},
  year = {2020},
  doi = {10.3390/s20236941},
}
Figure