Diverse Sequential Subset Selection for Supervised Video Summarization

Boqing Gong, Wei-Lun Chao, Kristen Grauman, Fei Sha

Introduction

Video summarization—the process of distilling a long video into a short, searchable sequence of keyframes—is traditionally treated as an unsupervised "subset selection" problem. Researchers usually hand-craft rules to prioritize properties like representativeness (covering the main content) and diversity (avoiding redundancy).

However, this traditional approach suffers from two major gaps:

  1. Human Subjectivity: "Quality" is highly subjective. Hand-tuned heuristics often fail to capture the nuanced logic humans use when editing a video.
  2. Temporal Blindness: Standard diversity models, like the Determinantal Point Process (DPP), treat items as a "bag of frames." They do not account for whether two similar frames are 5 seconds apart or 50 minutes apart. In video, temporal context is critical; two similar shots might be redundant if adjacent, but both might be necessary if they bookend a long narrative arc.

To address these issues, this paper transforms video summarization from a hand-crafted, unsupervised task into a supervised learning problem. By introducing the Sequential Determinantal Point Process (seqDPP), the authors create a model that learns from human-edited summaries how to select diverse, representative frames while respecting the chronological flow of video data.

Methodology

The authors propose a supervised framework consisting of three main components designed to capture both human subjectivity and temporal dynamics.

1. Oracle Summary Generation

Because different humans summarize the same video differently, training a supervised model requires a robust ground truth. The authors developed a greedy algorithm to synthesize a single "oracle" summary that maximizes the agreement (F-score) across multiple human annotators.

Comparison of Inter-user and Oracle-user F-score, Precision, and Recall.

As shown above, the generated oracle summaries consistently achieve higher mean F-scores, precision, and recall compared to average inter-user agreement, providing a strong, representative target for the supervised model.

2. Sequential DPP (seqDPP)

The core innovation of the paper is the seqDPP. While a standard DPP models the probability of a subset based on a global kernel matrix, seqDPP breaks the video into disjoint segments.

Graphical model representation of the Sequential Determinantal Point Process (seqDPP).

The selection of frames in the current segment depends on the frames selected in the immediate past segment. This Markov-like structure allows the model to enforce diversity locally while permitting similar content to reappear if enough time has passed, acknowledging that time is a primary dimension of video data.

3. Learned Representations

Instead of using fixed similarity metrics, the authors use Linear Embeddings and Nonlinear Hidden Representations (Neural Networks) to learn the DPP kernel. This allows the system to learn complex "meta-cues"—like camera motion or object placement—that signal a frame's importance, rather than relying solely on low-level visual similarity.

Datasets and Evaluation

The model was evaluated on three distinct benchmarks:

  • Open Video Project (OVP): 50 videos with diverse subject matter.
  • YouTube Dataset: 39 videos (excluding cartoons) featuring more dynamic, consumer-grade content.
  • Kodak Consumer Video Dataset: 18 videos representing typical personal home movies.

For OVP and YouTube, the authors used 5 human annotations per video to create their "oracle" training targets.

Key Insights and Results

The seqDPP model consistently outperformed both unsupervised methods and the standard "bag-of-frames" DPP:

  • Superior Performance: On the OVP Dataset, the seqDPP with a neural network backend achieved an F-score of 77.7%, significantly higher than the standard DPP (70.8%) and the best unsupervised method, VSUMM (70.3%).
  • Diversity vs. Redundancy: Qualitative results showed that while unsupervised methods often "collapsed" and picked too many frames from high-variance segments (like close-ups of people talking), seqDPP maintained a steady pulse throughout the video. It ensured the end of the video was as well-represented as the beginning.
  • Nonlinearity Matters: Using a neural network to learn the kernel parameters consistently beat simple linear transformations, suggesting that the "rules" of good summarization are non-linear and complex.
  • Learning "Meta-Cues": The authors argue that a supervised model doesn't just learn what specific objects look like; it learns the visual signatures of a good summary (e.g., how a cinematographer uses a zoom or a transition), which can generalize across videos with completely different subjects.

Limitations and Future Work

While highly effective, the approach has a few limitations:

  • Annotation Dependency: The method requires high-quality human summaries for training, which are expensive and time-consuming to produce.
  • Low-Level Features: The study primarily used Fisher Vectors (based on SIFT). Incorporating higher-level semantic features from deep Convolutional Neural Networks (CNNs) could further improve performance.
  • Long-Range Dependencies: While seqDPP captures dependencies between adjacent segments, it does not explicitly model very long-range dependencies (e.g., a character returning at the very end of a movie). Exploring more complex graphical models or recurrent architectures remains an exciting avenue for future research.

Summary updated

Abstract

Video summarization is a challenging problem with great application potential. Whereas prior approaches, largely unsupervised in nature, focus on sampling useful frames and assembling them as summaries, we consider video summarization as a supervised subset selection problem. Our idea is to teach the system to learn from human-created summaries how to select informative and diverse subsets, so as to best meet evaluation metrics derived from human-perceived quality. To this end, we propose the sequential determinantal point process (seqDPP), a probabilistic model for diverse sequential subset selection. Our novel seqDPP heeds the inherent sequential structures in video data, thus overcoming the deficiency of the standard DPP, which treats video frames as randomly permutable items. Meanwhile, seqDPP retains the power of modeling diverse subsets, essential for summarization. Our extensive results of summarizing videos from 3 datasets demonstrate the superior performance of our method, compared to not only existing unsupervised methods but also naive applications of the standard DPP model.
BibTeX
@inproceedings{gong2014diverse,
  author = {Boqing Gong and Wei-Lun Chao and Kristen Grauman and Fei Sha},
  title = {{Diverse Sequential Subset Selection for Supervised Video Summarization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
}
Figure