A bottom-up summarization algorithm for videos in the wild

Gang Pan, Yaoxian Zheng, Rufei Zhang, Zhenjun Han, Di Sun, Xingming QuEURASIP J. Adv. Signal Process 2019 · DOI

Introduction

Video summarization—the process of distilling a lengthy video into a concise version that retains its essential narrative—has become increasingly critical with the rise of unedited "videos in the wild." Traditional methods often approach this as a global optimization problem, attempting to select a set of frames that best represent the entire video simultaneously. However, this global perspective frequently results in "jumpy" summaries that fail to capture the temporal continuity and flow of human-centric storytelling. This paper proposes a "bottom-up" alternative that prioritizes local context and continuous actions through a greedy clip-growing strategy.

Methodology

The proposed algorithm utilizes a four-stage pipeline designed to transition from raw high-dimensional data to a coherent narrative summary.

  1. Pre-processing: To manage computational complexity, the system employs Singular Value Decomposition (SVD) to reduce 120x160 grayscale frames into 720D feature vectors.
  2. Video Oversegmentation: The video is clustered into $K$ compact local clips. This clustering uses a distance metric that balances visual feature similarity ($ds$) with temporal proximity ($dp$), ensuring clips are both visually and chronologically coherent.
  3. Importance Measure (Energy): Each frame is assigned an "Energy" value based on a dual-factor function:
  4. Dissimilarity ($Ed$): Detects action and motion by measuring pixel changes relative to subsequent frames.
  5. Representativeness ($Er$): Identifies stable, informative scenes by measuring similarity to neighbors within a sliding window.
  6. Clip Growing: This core stage starts with the highest-energy clip and expands the summary greedily based on user-defined target lengths and texture preferences.
Flowchart of the proposed bottom-up video summarization pipeline.

The Clip Growing Strategy

The defining novelty of this approach is the shift from global solvers to local growth. Instead of selecting disparate frames, the algorithm "grows" segments. It evaluates neighboring frames to decide whether to append them to an existing clip or initiate a new seed elsewhere.

The video clip growth decision process, illustrating how segments are expanded or new seeds selected.

This decision process is governed by a weight coefficient ($\Theta$), which allows for user customization. By adjusting this parameter, users can dictate the "texture" of the output—choosing between a highlight reel of rapid cuts or a more narrative summary featuring longer, continuous segments.

Results and Evaluation

The algorithm was benchmarked using the SumMe dataset, which contains 25 "in the wild" videos of various events and sports, accompanied by human-annotated importance scores.

In comparative testing against baselines like uniform sampling and the "Superframe" method, the proposed algorithm performed best or second-best in 21 out of 24 test cases. Notably, it achieved an average performance of 57% relative to the human upper bound, outperforming the Superframe method by 5%. The system proved particularly effective on static camera footage where the energy function could easily distinguish between background stability and the motion of primary subjects.

Conclusion

By focusing on local continuity rather than global representativeness, this bottom-up approach generates summaries that feel more natural to human viewers. While the current model faces challenges regarding the subjectivity of "energy" (where camera shake might be misinterpreted as high importance) and hyperparameter sensitivity, it provides a flexible framework for future enhancements. Future work may involve integrating semantic features like face detection and automating the estimation of optimal cluster counts.

Summary updated

BibTeX
@article{pan2019bottom,
  author = {Gang Pan and Yaoxian Zheng and Rufei Zhang and Zhenjun Han and Di Sun and Xingming Qu},
  title = {{A bottom-up summarization algorithm for videos in the wild}},
  journal = {EURASIP J. Adv. Signal Process.},
  year = {2019},
  doi = {10.1186/s13634-019-0611-y},
}
Figure