Abstract
Existing video summarization approaches mainly concentrate on sequential or structural characteristic of video data. However, they do not pay enough attention to the video summarization task itself. In this paper, we propose a meta learning method for performing task-driven video summarization, denoted by MetaL-TDVS, to explicitly explore the video summarization mechanism among summarizing processes on different videos. Particularly, MetaL-TDVS aims to excavate the latent mechanism for summarizing video by reformulating video summarization as a meta learning problem and promote generalization ability of the trained model. MetaL-TDVS regards summarizing each video as a single task to make better use of the experience and knowledge learned from processes of summarizing other videos to summarize new ones. Furthermore, MetaL-TDVS updates models via a two-fold back propagation which forces the model optimized on one video to obtain high accuracy on another video in every training step. Extensive experiments on benchmark datasets demonstrate the superiority and better generalization ability of MetaL-TDVS against several state-of-the-art methods.
1. Introduction
As digital video content continues to proliferate, automatic video summarization—the process of extracting the most informative frames while removing redundancy—has become a critical task for efficient storage and browsing. Traditional supervised deep learning approaches typically treat this as a structured prediction or sequence-to-sequence problem. While effective at capturing temporal dependencies within a specific dataset, these methods often struggle to generalize to unseen videos because they focus on data-level patterns rather than the underlying task of summarization.
To address this, we introduce MetaL-TDVS (Meta Learning for Task-Driven Video Summarization). By framing summarization as a meta-learning problem, our approach explicitly learns the latent mechanism of summarization, allowing it to leverage knowledge from previous videos to summarize new ones more effectively.
2. The MetaL-TDVS Framework
The core philosophy of MetaL-TDVS is to treat the summarization of each individual video as a distinct "task." The framework employs a base learner consisting of a bidirectional Long Short-Term Memory (vsLSTM) network and a Multi-Layer Perceptron (MLP) to output frame-level importance scores.
The innovation lies in the training loop, which utilizes a two-stage meta-learning update (two-fold backpropagation), as shown in Figure 1. In each iteration, the model parameters are updated based on the loss from a first video, then evaluated on a second, different video. The final parameter update for the iteration is derived from the performance on this second task, forcing the model to learn transferable representations.
3. Experimental Evaluation
We evaluated MetaL-TDVS using a rigorous transfer learning setup across four benchmark datasets: SumMe, TVSum, YouTube, and OVP. In this setup, the model is trained on three datasets and tested on the fourth, ensuring it is evaluated on entirely unseen data distributions. The model was tested with both deep features (GoogLeNet) and traditional shallow features (e.g., SIFT, HOG) to assess its robustness.
4. Results and Discussion
MetaL-TDVS achieved state-of-the-art results, reaching F-scores of 58.2% on TVSum and 44.1% on SumMe using deep features. The framework also demonstrated high efficiency, processing videos at approximately 1969 frames per second.
A critical insight from our experiments is the advantage of the successive meta-learning approach over standard simultaneous training (batching). As illustrated in Figure 2, the meta-learning variants (e.g., "twoMax") consistently outperform simultaneous training ("simu") across various learning rates. This confirms that the task-to-task transfer mechanism is the primary driver of the model's superior generalization capabilities.
5. Conclusion
MetaL-TDVS represents a shift from data-centric to task-centric learning in video summarization. By explicitly modeling the summarization mechanism through meta-learning, we achieve better generalization and state-of-the-art performance. Future work will focus on developing more sophisticated base learner architectures and exploring advanced meta-learning frameworks to further enhance the model's ability to capture complex video structures.
Summary updated