Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization

Anas Anwarul Haq Khan, Utkarsh Verma, Prateek Chanda, Ganesh RamakrishnanarXiv 2025 · DOI · arXiv

Abstract

Video summarization is critical for modern content consumption but remains computationally demanding for large-scale Vision-Language Models (VLMs). We present DEEVISum, a framework employing Multi-Stage Knowledge Distillation (MSKD) and an Early Exit (EE) mechanism to optimize performance and efficiency. By utilizing a 10B parameter mentor model to bridge the gap between a 28B teacher and a 3B student, and implementing confidence-based early termination, DEEVISum achieves state-of-the-art results on standard benchmarks while reducing inference time by 21%.

Introduction

The proliferation of short-form video platforms like TikTok and YouTube Shorts has heightened the need for automated video summarization—the process of condensing long-form content into concise, engaging highlights. While massive Vision-Language Models (VLMs) with billions of parameters excel at semantic understanding, their deployment is often hindered by high computational costs and slow inference speeds.

Researchers from IIT Bombay address these challenges with DEEVISum (Distilled Early-Exit VIsion-language model for Summarization). The approach tackles two primary bottlenecks: the "knowledge gap" encountered when distilling massive models into smaller ones, and the "computational waste" associated with processing every video frame with uniform intensity regardless of its importance.

Methodology

Multi-Stage Knowledge Distillation (MSKD)

A significant hurdle in model compression is the representational disparity between a massive "Teacher" model (e.g., 28B parameters) and a compact "Student" model (e.g., 3B parameters). Direct distillation often leads to suboptimal performance because the complexity jump is too steep.

DEEVISum introduces a Mentor model (10B parameters) to act as an intermediate bridge. In this hierarchical setup, the Teacher trains the Mentor, and the Student subsequently learns from both the Teacher and the Mentor simultaneously. This multi-stage process provides a smoother learning trajectory, allowing the 3B student to capture nuances that would otherwise be lost.

The Multi-Stage Knowledge Distillation (MSKD) process using a 10B mentor model to bridge the gap between 28B and 3B models.

Early Exit (EE) Mechanism

To reduce computational waste, the researchers implemented a MuE-style Early Exit strategy. "Exit heads" are integrated at various depths within the model's decoder. During inference, the model calculates a confidence score based on the cosine similarity between its current hidden state and a learned "summary prototype."

If the confidence score exceeds a predefined threshold, the model terminates processing for that frame and returns the result immediately. This allows the model to bypass deeper, more expensive layers for frames that are easily classified as unimportant or highly relevant.

The Early Exit architecture showing decoder exits and the confidence-based return condition.

Multimodal Prompting

Beyond visual frames, DEEVISum leverages a rich set of multimodal inputs. The model is prompted with the video title (T), transcript (Tr), and audio-derived features including gender-emotion classification (Ge) and speaker diarization (Sd). This holistic approach ensures the model understands the narrative context and emotional beats of the video.

Experimental Results

The framework was evaluated on the TVSum and SumMe benchmarks, as well as a curated dataset of complex, long-form videos.

  • Performance Gains: The MSKD-enhanced PaLI-Gemma2-3B model achieved a state-of-the-art F1 score of 61.1 on TVSum. This represents a 1.33% absolute improvement over the baseline 3B model, significantly outperforming standard single-stage distillation (0.5% gain).
  • Efficiency Gains: The Early Exit mechanism achieved a 21% reduction in inference time (from approximately 715s to 507s for the test set) with a negligible 1.3-point drop in F1 score.
  • Modality Impact: Incorporating transcripts (T + Tr) boosted the F1 score from 59.2 to 60.4. While emotion cues (Ge) provided further marginal gains, speaker diarization (Sd) introduced slight noise, suggesting that precise speaker boundaries are difficult to model in this context.

Conclusion

DEEVISum demonstrates that sophisticated distillation and adaptive inference can make high-performance video summarization accessible on smaller hardware. By bridging the knowledge gap with a mentor model and exiting early on confident predictions, the researchers have created a model that is both faster and more accurate than larger predecessors. Future research will focus on refining these mechanisms for even more complex, long-form video understanding.

Summary updated

Abstract

We introduce DEEVISum (Distilled Early Exit Vision language model for Summarization), a lightweight, efficient, and scalable vision language model designed for segment wise video summarization. Leveraging multi modal prompts that combine textual and audio derived signals, DEEVISum incorporates Multi Stage Knowledge Distillation (MSKD) and Early Exit (EE) to strike a balance between performance and efficiency. MSKD offers a 1.33% absolute F1 improvement over baseline distillation (0.5%), while EE reduces inference time by approximately 21% with a 1.3 point drop in F1. Evaluated on the TVSum dataset, our best model PaLI Gemma2 3B + MSKD achieves an F1 score of 61.1, competing the performance of significantly larger models, all while maintaining a lower computational footprint. We publicly release our code and processed dataset to support further research.
BibTeX
@misc{khan2025early,
  author = {Anas Anwarul Haq Khan and Utkarsh Verma and Prateek Chanda and Ganesh Ramakrishnan},
  title = {{Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization}},
  year = {2025},
  eprint = {2504.21831},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.21831},
}
Figure