A Statistical Framework for Video Skimming Based on Logical Story Units and Motion Activity

Sergio Benini, Pierangelo Migliorati, Riccardo LeonardiCBMI 2007 · OpenReview

TL;DR

This paper introduces a novel video skimming technique that leverages Hidden Markov Models (HMMs) to represent logical story units and motion activity to prioritize informative video segments. By modeling story structure and shot dynamics, the method generates condensed video summaries that aim to preserve content representational value and user enjoyability.

What Problem Does This Paper Address?

The explosion of digital video content has outpaced our ability to efficiently access and consume it. Video summarization, particularly video skimming (creating condensed video clips), is crucial for managing this information overload. While previous methods focused on extracting highlights or static keyframes, this paper tackles the challenge of creating dynamic summaries that capture the narrative flow and informative content of the original video, especially for unscripted content like home videos and sports, as well as scripted content like movies and news. The core problem is to intelligently select and sequence video segments to create a concise yet representative and engaging summary.

Methods

The proposed method combines two key ideas: Logical Story Units (LSUs) and Motion Activity.

  1. Logical Story Units (LSUs): The paper first segments videos into LSUs, which are defined as contiguous sequences of shots sharing a common semantic thread. This is achieved by building a Scene Transition Graph (STG) and identifying connected subgraphs after removing cut-edges. Each LSU is then modeled using a Hidden Markov Model (HMM).
  2. HMM States: Represent clusters of visually similar and temporally close shots within an LSU.
  3. HMM Transitions: Model the temporal dependencies and "shot patterns" within an LSU (e.g., dialogue sequences).
  4. HMM Observations: Individual video shots.
  5. Motion Activity: To determine the "informativeness" of each shot, the paper quantifies its motion activity. This is done by:
  6. Extracting motion vectors (MVs) from MPEG-compressed video (specifically from P-frames) for computational efficiency.
  7. Filtering these MVs to remove noise and improve reliability.
  8. Calculating the standard deviation of MV magnitudes within inter-coded macro-blocks as a measure of motion intensity for each shot. This intensity is interpreted as a proxy for the "entropy" or "content informativeness" of the shot.

Skim Generation: The video skim is generated by creating an observation sequence from the HMMs representing each LSU. Shots with higher motion intensity are assigned a higher probability of being selected as observations, thus prioritizing more dynamic and potentially informative segments. The selection process also considers representativeness (proportional duration per LSU) and coverage (including all LSUs).

Datasets

The experiments were conducted on a dataset comprising approximately four hours of video, including over two thousand shots. The dataset consists of various genres:

  • Portuguese News
  • Notting Hill (movie)
  • A Beautiful Mind (movie)
  • Pulp Fiction (movie)
  • Camilo & Filho (soap)
  • Riscos (soap)
  • Misc. (basket/soap/quiz)
  • Don Quixotte (cartoon)
  • Music Show (music)

The paper does not introduce a new benchmark dataset but uses this diverse set to evaluate the proposed method.

Key Insights and Ideas

  • Motion as a Proxy for Informativeness: The core insight is that motion intensity in a video shot can serve as a reliable indicator of its "content informativeness" or "entropy." More dynamic shots are more likely to contain key events or actions.
  • HMMs for Narrative Structure: Using HMMs to model LSUs effectively captures both the semantic content (via states) and the temporal flow (via transitions) of story segments, providing a robust framework for understanding video narrative.
  • Balancing Informativeness and Enjoyability: The method aims to balance the selection of informative shots (high motion) with the overall coherence and enjoyability of the skim. By allowing lower-motion shots to be selected with lower probability, especially for longer skims, it avoids discarding potentially important but less dynamic content.
  • Efficient Motion Extraction: Leveraging motion vectors from MPEG-compressed streams significantly reduces computational cost compared to pixel-based motion analysis.

Key Results

The paper reports on experiments evaluating the proposed method using "informativeness" and "enjoyability" metrics, as defined in prior work.

  • Enjoyability: Average normalized scores for enjoyability were around 72% for 10% skim ratio and 80% for 25% skim ratio.
  • Informativeness: Average normalized scores for informativeness were around 68% for 10% skim ratio and 81% for 25% skim ratio. These results are presented as comparable to other state-of-the-art methods, achieved on a diverse dataset. Notably, the results suggest that even when shots are not presented in their original temporal order (similar to movie trailers), the skim can still preserve informativeness and viewer engagement, provided the visual content remains coherent.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the skim generation does not strictly adhere to the original shot order, which might be a concern for some applications, though results suggest viewers are not significantly disturbed by this.
  • While motion activity is a good proxy, it doesn't capture all aspects of "informativeness" (e.g., semantic meaning of static objects or dialogue).
  • Future Work:
    • The authors suggest that further work could explore incorporating other cues (audio, text) to enhance the informativeness and accuracy of the skims.
  • Investigating more sophisticated HMM state definitions or transition models could further refine the representation of story units.

Summary updated

Abstract

In this work we present a method for video skimming based on hidden Markov Models (HMMs) and motion activity. Specifically, a set of HMMs is used to model subsequent logical story units, where the HMM states represent different visual-concepts, the transitions model the temporal dependencies in each story unit, and stochastic observations are given by single shots. The video skim is generated as an observation sequence, where, in order to privilege more informative segments for entering the skim, dynamic shots are assigned higher probability of observation. The effectiveness of the method is demonstrated on a video set from different kinds of programmes, and results are evaluated in terms of metrics that measure the content representational value of the obtained video skims.
BibTeX
@inproceedings{benini2007statistical,
  author = {Sergio Benini and Pierangelo Migliorati and Riccardo Leonardi},
  title = {{A Statistical Framework for Video Skimming Based on Logical Story Units and Motion Activity}},
  booktitle = {CBMI},
  year = {2007},
  url = {https://openreview.net/forum?id=ps4F3Sx4ri},
}
Figure