Abstract
Keyframe extraction aims to summarize a video's semantics using a minimal number of frames. This paper introduces LMSKE (Large Model based Sequential Keyframe Extraction), a three-stage framework for video summarization. First, the video is segmented into consecutive shots using TransNetV2, and semantic visual features are extracted for each frame using CLIP. Second, an adaptive clustering algorithm determines the optimal number of candidate keyframes per shot. Finally, redundancy elimination is performed within each shot to produce a concise, sequential summary. Experiments on a newly curated benchmark demonstrate that LMSKE outperforms state-of-the-art competitors, achieving an average F1 score of 0.5311, fidelity of 0.8141, and a compression ratio of 0.9922.
Introduction
In an era dominated by short-form and long-form video content on platforms like TikTok and YouTube, the ability to extract meaningful summaries is critical. Keyframe extraction—the process of selecting a handful of frames that capture the essence of a video—is a foundational task for storage, retrieval, and analysis. However, traditional methods often struggle to balance semantic depth with temporal order.
Existing techniques typically fall into four categories: uniform sampling (efficient but semantically poor), clustering-based (semantically strong but temporally disjointed), comparison-based (threshold-sensitive), and shot-based (temporally consistent but often over-simplified). LMSKE bridges these gaps by utilizing foundation models and an automated mechanism to determine shot-specific complexity.
Methodology
The LMSKE framework operates in three distinct stages:
1. Shot Segmentation and Feature Extraction
The video is first processed by TransNetV2 to detect shot transitions, breaking the content into logical segments. Each frame within a shot is then passed through CLIP to generate a 768-dimensional semantic vector. Unlike traditional pixel-based features, CLIP captures high-level concepts, allowing the system to understand what is happening in a scene rather than just how the pixels change.
2. Adaptive Clustering
To avoid the common pitfall of manually defining the number of clusters ($k$), LMSKE uses an adaptive approach. It evaluates various cluster counts and employs the Silhouette Coefficient (SC) to find the "sweet spot"—the number of clusters that maximizes the distinctness of the frames. Frames closest to these optimal cluster centers are selected as candidate keyframes.
3. Redundancy Elimination
The system filters out uninformative frames (e.g., solid colors) using HSV color histograms. It then performs a final redundancy check within each shot, removing near-duplicate candidates to ensure the summary is as lean as possible while maintaining the narrative flow.
Experimental Results
The researchers evaluated LMSKE using TVSum20, a benchmark derived from the TVSum dataset containing 20 videos across 10 diverse categories. LMSKE was compared against several established methods, including VSUMM and K-Means.
Performance Metrics: * F1 Score: 0.5311 (Superior balance of precision and recall) * Fidelity: 0.8141 (High representativeness of the original video) * Compression Ratio: 0.9922 (Highly efficient summarization)
In a qualitative case study involving a video on grooming a dog, LMSKE matched 12 out of 13 benchmark keyframes. Its closest competitor, an Inception-based method, matched 11 frames but required a total of 20 frames, highlighting LMSKE's superior efficiency and accuracy.
Conclusion
LMSKE demonstrates that leveraging large-scale foundation models like CLIP and TransNetV2 significantly improves the quality and efficiency of video summarization. By prioritizing semantics and maintaining temporal sequence through shot-based processing, it provides a robust framework for modern video analysis. Future work will address cross-shot redundancy and further optimize the computational overhead of using multiple large models.
Summary updated