Co-Regularized Deep Representations for Video Summarization

Olivier Morère, Hanlin Goh, Antoine Veillard, Vijay Chandrasekhar, Jie Lin

Abstract

Compact keyframe-based video summaries are a popular way of generating viewership on video sharing platforms. Yet, creating relevant and compelling summaries for arbitrarily long videos with a small number of keyframes is a challenging task. We propose a comprehensive keyframe-based summarization framework combining deep convolutional neural networks and restricted Boltzmann machines. An original co-regularization scheme is used to discover meaningful subject-scene associations. The resulting multimodal representations are then used to select highly-relevant keyframes. A comprehensive user study is conducted comparing our proposed method to a variety of schemes, including the summarization currently in use by one of the most popular video sharing websites. The results show that our method consistently outperforms the baseline schemes for any given amount of keyframes both in terms of attractiveness and informativeness. The lead is even more significant for smaller summaries.

Introduction

As digital video content continues to grow exponentially, the demand for compact, representative summaries—typically a small set of keyframes—has become critical for platforms like YouTube or Dailymotion. Traditional summarization methods often rely on low-level visual features such as color histograms or motion vectors, or simple clustering techniques like k-means. However, these approaches frequently fail to capture the high-level semantic meaning of a frame. A k-means approach might select a blurry frame simply because its color profile is unique, whereas a human viewer would prefer a clear, meaningful shot of a subject in its environment. This paper addresses the gap by proposing a system that understands both the "what" (subject) and the "where" (scene) of a video frame.

Methodology: Co-Regularized RBMs

The proposed framework utilizes a multi-stage pipeline to bridge the semantic gap. First, features are extracted using two Deep Convolutional Neural Networks (DCNNs): a VGG-ILSVRC-2014-D model trained on ImageNet for object recognition, and a Places-CNN model for scene classification.

The core innovation lies in the use of co-regularized Restricted Boltzmann Machines (RBMs). These RBMs are trained concurrently with a penalty term that forces coordination between the subject and scene descriptors. For example, if the subject RBM identifies a "Polar Bear," the co-regularization encourages the scene RBM to associate it with an "Iceberg." This process effectively "binds" subjects to their natural contexts, creating a multimodal representation that captures meaningful associations.

Architecture of the co-regularized RBM framework and the frame descriptor calculation.

Keyframes are selected by identifying frames that produce the maximum response in these joint representations. A balance parameter, $\alpha$, allows for the generation of summaries that can be tuned to be more subject-centric or scene-centric.

Key Insights and Tunability

A significant advantage of this approach is the ability to provide tunable summaries. Unlike "black-box" algorithms, this method offers a "knob" ($\alpha$) to adjust the focus of the summary. A biologist might prefer a high $\alpha$ value to focus on animal subjects, while a cinematographer might prefer a lower value to emphasize landscape vistas.

Visual comparison of summaries generated with different alpha values, illustrating the shift from subject-centric to scene-centric selection.

Furthermore, visualization of the hidden layers reveals that the RBM units learn recognizable concepts autonomously. Units often fire for specific subject-scene pairs (e.g., sharks in oceans) without explicit supervision, discovering these correlations through the co-regularization of DCNN features.

Experimental Results

The framework was evaluated using the BBC nature documentary series Planet Earth, a challenging dataset featuring diverse wildlife in distinct natural habitats. A user study compared the proposed method against uniform sampling, k-means clustering, and the production algorithm used by Dailymotion.

The results demonstrated a clear preference for the proposed method:

  • User Preference: Testers preferred the proposed method 75% to 97% of the time over baseline schemes in terms of both attractiveness and informativeness.
  • Industry Comparison: The method outperformed Dailymotion’s existing algorithm by more than a 3-to-1 margin.
  • Efficiency: The performance lead was most significant for very small summaries (e.g., 4 keyframes), indicating the model's superior ability to identify the most representative moments.

Conclusion

While the current model treats frames independently and relies on pre-trained DCNNs, it represents a significant step forward in semantic video summarization. Future work may involve incorporating temporal continuity for narrative-heavy content or extending the co-regularization approach to other modalities, such as binding faces to activities or audio cues to visual events.

Summary updated

Abstract

Compact keyframe-based video summaries are a popular way of generating viewership on video sharing platforms. Yet, creating relevant and compelling summaries for arbitrarily long videos with a small number of keyframes is a challenging task. We propose a comprehensive keyframe-based summarization framework combining deep convolutional neural networks and restricted Boltzmann machines. An original co-regularization scheme is used to discover meaningful subject-scene associations. The resulting multimodal representations are then used to select highly-relevant keyframes. A comprehensive user study is conducted comparing our proposed method to a variety of schemes, including the summarization currently in use by one of the most popular video sharing websites. The results show that our method consistently outperforms the baseline schemes for any given amount of keyframes both in terms of attractiveness and informativeness. The lead is even more significant for smaller summaries.
BibTeX
@inproceedings{morere2015co,
  author = {Olivier Morère and Hanlin Goh and Antoine Veillard and Vijay Chandrasekhar and Jie Lin},
  title = {{Co-Regularized Deep Representations for Video Summarization}},
  booktitle = {ICIP},
  year = {2015},
  url = {https://openreview.net/forum?id=ILj4ORs4OoB},
}
Figure