Abstract
Traditional video summarization techniques are often constrained by the high cost and inherent subjectivity of human-annotated datasets. This paper presents a novel self-supervised framework that redefines video summarization as a Natural Language Processing (NLP) task. By leveraging Large Language Models (LLMs) to synthesize frame-level captions into a global narrative and employing semantic matching via Siamese-BERT, the proposed method identifies key segments without requiring manual labels. A core contribution is the Preserving Diversity Loss (PDL), an adaptive mechanism that adjusts to the visual variance of different datasets. Experimental results demonstrate that this approach achieves state-of-the-art performance on the SumMe benchmark and competitive results on TVSum, while maintaining high computational efficiency.
Introduction
Video summarization aims to condense long video sequences into short, representative versions that preserve the essential narrative. Historically, this field has been dominated by supervised learning models. However, these models require massive amounts of human-labeled data, which is not only expensive to produce but also reflects the subjective biases of the annotators. Unsupervised methods, while more scalable, often fail to capture the high-level semantic context, focusing instead on low-level visual features like color or motion.
This research bridges the gap between computer vision and NLP by treating video frames as semantic units. By converting visual data into descriptive text, the framework utilizes the advanced reasoning capabilities of LLMs to determine "importance" based on narrative coherence rather than just visual saliency.
Methodology
The proposed framework operates through a multi-stage pipeline designed to transform visual sequences into a semantic matching problem.
1. Frame Captioning and LLM Synthesis
The process begins by downsampling the video and generating descriptive captions for individual frames using a pre-trained Generative Image-to-text Transformer (GIT). These captions are then aggregated and fed into GPT-4. The LLM acts as a high-level reasoning engine, synthesizing the disparate frame descriptions into a single, coherent text summary that captures the overall essence of the video.
2. Semantic Matching via Siamese-BERT
To determine which frames belong in the final summary, the model employs a Siamese-BERT architecture. It calculates the semantic distance between each individual frame's caption and the global summary generated by the LLM. Frames whose captions align closely with the global narrative are assigned higher importance scores. This approach allows the model to benefit from the vast world-knowledge embedded in LLMs without requiring the LLM to process the video directly.
3. Preserving Diversity Loss (PDL)
A significant challenge in video summarization is the varying level of redundancy across different types of content. For instance, a cooking video (SumMe) might have many similar-looking scenes, whereas a YouTube compilation (TVSum) is highly diverse. The authors introduced the Preserving Diversity Loss (PDL) to address this. PDL calculates a diversity score ($D$) based on scene transitions and dynamically adjusts the model's regularization. This ensures the summary covers different "chapters" of the video rather than clustering around a single interesting but repetitive event.
4. Optimization and Selection
The final selection of frames is treated as a knapsack problem, where the model selects a subset of scenes that maximize the importance scores while staying within a predefined time constraint (typically 15% of the original video length).
Experimental Results
The framework was evaluated on the SumMe and TVSum datasets using rank correlation coefficients (Kendall’s $\tau$ and Spearman’s $\rho$) to compare model-generated scores against human rankings.
- SumMe: The model achieved state-of-the-art (SOTA) performance for unsupervised methods, reaching a Spearman’s $\rho$ of 0.138. This outperformed several supervised benchmarks, proving that semantic text matching can rival human-guided training.
- TVSum: The model achieved the second-best results in the unsupervised category with a Spearman’s $\rho$ of 0.174. The authors noted that their model maintained consistent hyperparameters, whereas some higher-performing competitors used dataset-specific tuning.
- Efficiency: Unlike attention-based models that scale quadratically ($O(d \cdot n^2)$), this linear semantic matching approach operates at $O(d \cdot n)$ complexity, making it highly suitable for long-form video content.
Discussion and Insights
The primary insight of this work is the effectiveness of "Video as Text." By abstracting visual data into language, the model can leverage the context-awareness of LLMs to perform zero-shot tasks. This is particularly evident in personalized summarization; a user can provide a specific query (e.g., "focus on the dog in the park"), and the LLM will adjust the global summary, naturally shifting the semantic matching process to favor relevant frames.
However, the method is currently limited by its reliance on static frame captions. It does not yet fully account for temporal dependencies or motion-based cues. Furthermore, the quality of the summary is inherently tied to the accuracy of the initial image-to-text model.
Conclusion
This research demonstrates a powerful shift toward language-guided, self-supervised video understanding. By combining the descriptive power of image-to-text models with the reasoning capabilities of LLMs and an adaptive diversity loss, the framework provides a robust, efficient, and highly interpretable solution for video summarization. Future developments will likely focus on integrating video-language models that can interpret temporal flow and motion directly, further refining the semantic bridge between vision and language.
Summary updated