Large Model based Sequential Keyframe Extraction for Video Summarization

Kailong Tan, Yuxiang Zhou, Qianchen Xia, Rui Liu, Yong Chen

Abstract

Keyframe extraction aims to summarize a video's semantics using a minimal number of frames. This paper introduces LMSKE (Large Model based Sequential Keyframe Extraction), a three-stage framework for video summarization. First, the video is segmented into consecutive shots using TransNetV2, and semantic visual features are extracted for each frame using CLIP. Second, an adaptive clustering algorithm determines the optimal number of candidate keyframes per shot. Finally, redundancy elimination is performed within each shot to produce a concise, sequential summary. Experiments on a newly curated benchmark demonstrate that LMSKE outperforms state-of-the-art competitors, achieving an average F1 score of 0.5311, fidelity of 0.8141, and a compression ratio of 0.9922.

Introduction

In an era dominated by short-form and long-form video content on platforms like TikTok and YouTube, the ability to extract meaningful summaries is critical. Keyframe extraction—the process of selecting a handful of frames that capture the essence of a video—is a foundational task for storage, retrieval, and analysis. However, traditional methods often struggle to balance semantic depth with temporal order.

Existing techniques typically fall into four categories: uniform sampling (efficient but semantically poor), clustering-based (semantically strong but temporally disjointed), comparison-based (threshold-sensitive), and shot-based (temporally consistent but often over-simplified). LMSKE bridges these gaps by utilizing foundation models and an automated mechanism to determine shot-specific complexity.

Methodology

The LMSKE framework operates in three distinct stages:

1. Shot Segmentation and Feature Extraction

The video is first processed by TransNetV2 to detect shot transitions, breaking the content into logical segments. Each frame within a shot is then passed through CLIP to generate a 768-dimensional semantic vector. Unlike traditional pixel-based features, CLIP captures high-level concepts, allowing the system to understand what is happening in a scene rather than just how the pixels change.

Feature extraction and representation within the LMSKE pipeline.

2. Adaptive Clustering

To avoid the common pitfall of manually defining the number of clusters ($k$), LMSKE uses an adaptive approach. It evaluates various cluster counts and employs the Silhouette Coefficient (SC) to find the "sweet spot"—the number of clusters that maximizes the distinctness of the frames. Frames closest to these optimal cluster centers are selected as candidate keyframes.

3. Redundancy Elimination

The system filters out uninformative frames (e.g., solid colors) using HSV color histograms. It then performs a final redundancy check within each shot, removing near-duplicate candidates to ensure the summary is as lean as possible while maintaining the narrative flow.

Experimental Results

The researchers evaluated LMSKE using TVSum20, a benchmark derived from the TVSum dataset containing 20 videos across 10 diverse categories. LMSKE was compared against several established methods, including VSUMM and K-Means.

Performance Metrics: * F1 Score: 0.5311 (Superior balance of precision and recall) * Fidelity: 0.8141 (High representativeness of the original video) * Compression Ratio: 0.9922 (Highly efficient summarization)

In a qualitative case study involving a video on grooming a dog, LMSKE matched 12 out of 13 benchmark keyframes. Its closest competitor, an Inception-based method, matched 11 frames but required a total of 20 frames, highlighting LMSKE's superior efficiency and accuracy.

A sequence of extracted keyframes illustrating the summarized narrative of a video.

Conclusion

LMSKE demonstrates that leveraging large-scale foundation models like CLIP and TransNetV2 significantly improves the quality and efficiency of video summarization. By prioritizing semantics and maintaining temporal sequence through shot-based processing, it provides a robust framework for modern video analysis. Future work will address cross-shot redundancy and further optimize the computational overhead of using multiple large models.

Summary updated

Abstract

Keyframe extraction aims to sum up a video's semantics with the minimum number of its frames. This paper puts forward a Large Model based Sequential Keyframe Extraction for video summarization, dubbed LMSKE, which contains three stages as below. First, we use the large model "TransNetV21" to cut the video into consecutive shots, and employ the large model "CLIP2" to generate each frame's visual feature within each shot; Second, we develop an adaptive clustering algorithm to yield candidate keyframes for each shot, with each candidate keyframe locating nearest to a cluster center; Third, we further reduce the above candidate keyframes via redundancy elimination within each shot, and finally concatenate them in accordance with the sequence of shots as the final sequential keyframes. To evaluate LMSKE, we curate a benchmark dataset and conduct rich experiments, whose results exhibit that LMSKE performs much better than quite a few SOTA competitors with average F1 of 0.5311, average fidelity of 0.8141, and average compression ratio of 0.9922.
BibTeX
@misc{tan2024large,
  author = {Kailong Tan and Yuxiang Zhou and Qianchen Xia and Rui Liu and Yong Chen},
  title = {{Large Model based Sequential Keyframe Extraction for Video Summarization}},
  year = {2024},
  eprint = {2401.04962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.04962},
}

Loading related content...

Figure