Parameter-free Video Segmentation for Vision and Language Understanding

Louis Mahon, Mirella Lapata

Introduction

Large Vision-Language Models (LVLMs) face significant challenges when processing long-form video content, such as movies or television episodes. The primary obstacle is the "memory wall"—the computational difficulty of processing hours of video end-to-end. Standard approaches often involve breaking video into smaller chunks using simple heuristics like camera cuts or fixed time windows. However, these methods frequently fail to capture narrative shifts, often splitting critical scenes in half or missing thematic transitions entirely.

To address these limitations, researchers have introduced MDLSeg, a parameter-free algorithm designed to segment long videos into semantically meaningful scenes. By leveraging the Minimum Description Length (MDL) principle, MDLSeg identifies narrative boundaries without requiring manual thresholds or domain-specific training.

MDLSeg: Segmentation via Compression

The core of MDLSeg is the application of the MDL principle to video features. This approach treats video segmentation as a data compression problem, effectively implementing a mathematical version of Occam’s Razor. The algorithm seeks the "simplest" description of video data by balancing two factors:

  1. Intra-scene Similarity: Frames within a single scene should have low variance.
  2. Model Complexity: The total number of scenes should be kept minimal.

MDLSeg calculates a "bitcost" for every potential partition of the video. It models feature vectors (extracted via CLIP) within each scene using a multivariate normal distribution. The total cost is the sum of the bits required to represent the frames given the scene's parameters and the bits needed to store those parameters. Using dynamic programming, the algorithm efficiently finds the globally optimal set of boundaries among millions of possibilities. This process is entirely parameter-free, removing the need for predefined scene counts or difference thresholds.

The "Pseudo-Screenplay" Pipeline

Beyond segmentation, the authors propose a modular "pseudo-screenplay" pipeline for downstream tasks like summarization. Instead of feeding raw video directly into an LLM, the system uses MDLSeg to define scene boundaries, extracts transcripts via WhisperX, identifies characters through facial recognition (cross-referenced with IMDB), and generates detailed visual descriptions.

Overview of the modular movie summarization pipeline using MDLSeg.

This modular approach allows the model to handle long-form content by processing it as a structured narrative rather than an unstructured stream of frames.

Experimental Evaluation

The effectiveness of MDLSeg was evaluated across several benchmarks, focusing on scene boundary accuracy, summarization quality, and long-form video question answering (VQA). MDLSeg demonstrated superior performance in scene detection, outperforming all baselines including "oracle" versions of other methods that were provided with the ground-truth number of scenes.

Performance comparison of MDLSeg against various segmentation baselines on OVSD and BBC datasets.

In movie summarization tasks, MDLSeg-based boundaries resulted in higher factual precision and better alignment with human summaries compared to uniform splitting or camera-cut detection. Furthermore, in retrieval-augmented VQA, the MDLSeg pipeline achieved 40.92% accuracy, significantly outperforming existing long-video models like Llama-vid (26.86%) and rivaling models specifically fine-tuned for the task.

Limitations and Future Work

While MDLSeg is highly effective, it currently relies solely on visual signals. Narrative shifts occurring through dialogue or music without visual changes may be missed. Future work aims to incorporate audio and transcribed text into the MDL cost function. Additionally, while the search algorithm is efficient, the initial extraction of high-dimensional CLIP embeddings for long movies remains a computational bottleneck.

Conclusion

MDLSeg represents a significant step forward in long-form video understanding. By reframing segmentation as a compression problem, it provides a robust, parameter-free method for identifying narrative structures, enabling Large Vision-Language Models to overcome the memory wall and process complex, hour-long content with greater accuracy.

Summary updated

Abstract

The proliferation of creative video content has driven demand for adapting language models to handle video input and enable multimodal understanding. However, end-to-end models struggle to process long videos due to their size and complexity. An effective alternative is to divide them into smaller chunks to be processed separately, and this motivates a method for choosing where the chunk boundaries should be. In this paper, we propose an algorithm for segmenting videos into contiguous chunks, based on the minimum description length principle, coupled with a dynamic programming search. The algorithm is entirely parameter-free, given feature vectors, not requiring a set threshold or the number or size of chunks to be specified. We show empirically that the breakpoints it produces more accurately approximate scene boundaries in long videos, compared with existing methods for scene detection, even when such methods have access to the true number of scenes. We then showcase this algorithm in two tasks: long video summarization, and retrieval-augmented video question answering. In both cases, scene breaks produced by our algorithm lead to better downstream performance than existing methods for video segmentation.
BibTeX
@misc{mahon2025parameter,
  author = {Louis Mahon and Mirella Lapata},
  title = {{Parameter-free Video Segmentation for Vision and Language Understanding}},
  year = {2025},
  eprint = {2503.01201},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.01201},
}
Figure