Abstract
Video summarization is critical for modern content consumption but remains computationally demanding for large-scale Vision-Language Models (VLMs). We present DEEVISum, a framework employing Multi-Stage Knowledge Distillation (MSKD) and an Early Exit (EE) mechanism to optimize performance and efficiency. By utilizing a 10B parameter mentor model to bridge the gap between a 28B teacher and a 3B student, and implementing confidence-based early termination, DEEVISum achieves state-of-the-art results on standard benchmarks while reducing inference time by 21%.
Introduction
The proliferation of short-form video platforms like TikTok and YouTube Shorts has heightened the need for automated video summarization—the process of condensing long-form content into concise, engaging highlights. While massive Vision-Language Models (VLMs) with billions of parameters excel at semantic understanding, their deployment is often hindered by high computational costs and slow inference speeds.
Researchers from IIT Bombay address these challenges with DEEVISum (Distilled Early-Exit VIsion-language model for Summarization). The approach tackles two primary bottlenecks: the "knowledge gap" encountered when distilling massive models into smaller ones, and the "computational waste" associated with processing every video frame with uniform intensity regardless of its importance.
Methodology
Multi-Stage Knowledge Distillation (MSKD)
A significant hurdle in model compression is the representational disparity between a massive "Teacher" model (e.g., 28B parameters) and a compact "Student" model (e.g., 3B parameters). Direct distillation often leads to suboptimal performance because the complexity jump is too steep.
DEEVISum introduces a Mentor model (10B parameters) to act as an intermediate bridge. In this hierarchical setup, the Teacher trains the Mentor, and the Student subsequently learns from both the Teacher and the Mentor simultaneously. This multi-stage process provides a smoother learning trajectory, allowing the 3B student to capture nuances that would otherwise be lost.
Early Exit (EE) Mechanism
To reduce computational waste, the researchers implemented a MuE-style Early Exit strategy. "Exit heads" are integrated at various depths within the model's decoder. During inference, the model calculates a confidence score based on the cosine similarity between its current hidden state and a learned "summary prototype."
If the confidence score exceeds a predefined threshold, the model terminates processing for that frame and returns the result immediately. This allows the model to bypass deeper, more expensive layers for frames that are easily classified as unimportant or highly relevant.
Multimodal Prompting
Beyond visual frames, DEEVISum leverages a rich set of multimodal inputs. The model is prompted with the video title (T), transcript (Tr), and audio-derived features including gender-emotion classification (Ge) and speaker diarization (Sd). This holistic approach ensures the model understands the narrative context and emotional beats of the video.
Experimental Results
The framework was evaluated on the TVSum and SumMe benchmarks, as well as a curated dataset of complex, long-form videos.
- Performance Gains: The MSKD-enhanced PaLI-Gemma2-3B model achieved a state-of-the-art F1 score of 61.1 on TVSum. This represents a 1.33% absolute improvement over the baseline 3B model, significantly outperforming standard single-stage distillation (0.5% gain).
- Efficiency Gains: The Early Exit mechanism achieved a 21% reduction in inference time (from approximately 715s to 507s for the test set) with a negligible 1.3-point drop in F1 score.
- Modality Impact: Incorporating transcripts (T + Tr) boosted the F1 score from 59.2 to 60.4. While emotion cues (Ge) provided further marginal gains, speaker diarization (Sd) introduced slight noise, suggesting that precise speaker boundaries are difficult to model in this context.
Conclusion
DEEVISum demonstrates that sophisticated distillation and adaptive inference can make high-performance video summarization accessible on smaller hardware. By bridging the knowledge gap with a mentor model and exiting early on confident predictions, the researchers have created a model that is both faster and more accurate than larger predecessors. Future research will focus on refining these mechanisms for even more complex, long-form video understanding.
Summary updated