Co-regularized deep representations for video summarization

Olivier Morère, Hanlin Goh, Antoine Veillard, Vijay Chandrasekhar, Jie Lin

TL;DR

This paper introduces a novel framework for video summarization that uses deep learning to "bind" subjects (like animals or objects) to their environments (scenes). By using a co-regularization scheme between two Restricted Boltzmann Machines (RBMs), the system learns to identify the most representative moments where a subject and its setting are both clearly present, significantly outperforming standard industry algorithms in user preference.

What Problem Does This Paper Address?

As video content explodes, the need for "compact summaries"—a few keyframes that represent a long video—is critical for user engagement on platforms like YouTube or Dailymotion.

The challenge is that traditional methods often rely on low-level visual features (like color histograms or motion) or simple clustering (k-means). These methods frequently miss the "story" or the high-level semantic meaning of a frame. For example, a k-means approach might pick a blurry frame because its colors are unique, whereas a human would prefer a clear shot of a lion in the savannah. The authors aim to bridge this gap by creating a system that understands both what is in the frame and where it is happening.

Methods

The authors propose a multi-stage pipeline:

  1. Feature Extraction (The "Eyes"): They use two Deep Convolutional Neural Networks (DCNNs). One is trained on ImageNet (VGG-ILSVRC-2014-D) to recognize 1,000 object categories. The other is trained on the Places-CNN dataset to recognize 205 different types of scenes (indoors and outdoors).
  2. Co-Regularized RBMs (The "Brain"): This is the core innovation. They feed the DCNN features into two Restricted Boltzmann Machines (RBMs) that are trained concurrently.
  3. Co-regularization: They introduce a penalty term that forces the "subject" RBM and the "scene" RBM to coordinate. If the subject RBM identifies a "Polar Bear," the co-regularization encourages the scene RBM to associate it with "Iceberg." This "binds" subjects to their natural contexts.
  4. Keyframe Selection: The final summary is generated by looking for frames that produce the maximum response in these joint representations. A balance parameter ($\alpha$) allows the user to tune the summary to be more "subject-centric" (focusing on the actors) or "scene-centric" (focusing on the landscape).

Datasets

The primary dataset used for evaluation is the BBC nature documentary series Planet Earth (11 episodes, each ~50 minutes long). This dataset is particularly challenging and relevant because it features diverse wildlife (subjects) in distinct natural habitats (scenes).

Key Insights and Ideas

  • Semantic Binding: The most powerful idea here is that a good summary isn't just about diversity; it's about meaningful associations. By forcing the model to learn that certain subjects belong in certain scenes, the system naturally gravitates toward high-quality, "textbook" shots.
  • Tunable Summaries: Unlike "black-box" algorithms, this method provides a "knob" ($\alpha$). If you are a biologist interested in the animals, you can turn up the subject-centricity. If you are a cinematographer interested in the vistas, you can turn up the scene-centricity.
  • Visualizing the Hidden Layers: The authors show that the RBM units actually learn recognizable concepts. For instance, one unit might consistently fire for "sharks" in "oceans," even though the RBM was never explicitly told those two things were related—it discovered the correlation through the co-regularization of the DCNN features.

Key Results

The researchers conducted a user study with eight testers, comparing their method against uniform sampling, k-means clustering, and the production algorithm used by Dailymotion.

  • Attractiveness & Informativeness: Users preferred the proposed method 75% to 97% of the time over the baselines.
  • Beating the Pros: Against Dailymotion’s existing algorithm, the proposed method was preferred by more than a 3-to-1 margin.
  • Efficiency in Compression: The performance gap was most noticeable when the summary was very small (e.g., only 4 keyframes). This suggests the model is excellent at picking the "best of the best" frames.

Limitations and Future Work

  • Temporal Blindness: The current model treats frames as independent images. It doesn't explicitly account for the "flow" of a story or temporal continuity, which might be important for narrative-heavy videos (like movies).
  • Pre-trained Dependence: The system relies heavily on the quality of the pre-trained DCNNs. If a video contains objects or scenes not covered by VGG or Places-CNN, the summarization quality might degrade.
  • Future Directions: The authors suggest that this co-regularization approach isn't limited to subjects and scenes; it could be generalized to other modalities, such as binding faces to specific activities or even audio cues to visual events.

Summary updated

Abstract

Compact keyframe-based video summaries are a popular way of generating viewership on video sharing platforms. Yet, creating relevant and compelling summaries for arbitrarily long videos with a small number of keyframes is a challenging task. We propose a comprehensive keyframe-based summarization framework combining deep convolutional neural networks and restricted Boltzmann machines. An original co-regularization scheme is used to discover meaningful subject-scene associations. The resulting multimodal representations are then used to select highly-relevant keyframes. A comprehensive user study is conducted comparing our proposed method to a variety of schemes, including the summarization currently in use by one of the most popular video sharing websites. The results show that our method consistently outperforms the baseline schemes for any given amount of keyframes both in terms of attractiveness and informativeness. The lead is even more significant for smaller summaries.
BibTeX
@inproceedings{morere2015co,
  author = {Olivier Morère and Hanlin Goh and Antoine Veillard and Vijay Chandrasekhar and Jie Lin},
  title = {{Co-regularized deep representations for video summarization}},
  booktitle = {ICIP},
  year = {2015},
  url = {https://openreview.net/forum?id=ILj4ORs4OoB},
}

Loading related content...

Figure