MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding

Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, Zheng Liu

TL;DR

This paper introduces MLVU, a new benchmark designed to rigorously evaluate how well multimodal large language models (MLLMs) understand long videos. MLVU addresses the limitations of existing benchmarks by using longer videos, a wider variety of content, and more challenging, diverse tasks, revealing that current MLLMs still struggle significantly with this complex domain.

What Problem Does This Paper Address?

The field of multimodal large language models (MLLMs) is rapidly advancing, with models capable of processing text, images, and even video. However, evaluating their ability to understand long videos (minutes to hours) is a significant challenge. Existing video understanding benchmarks often suffer from:

  • Short video lengths: Most benchmarks use videos that are only a few seconds long, which doesn't reflect real-world scenarios or test the models' ability to retain information over extended periods.
  • Lack of diversity: Benchmarks tend to focus on a single type of video (e.g., egocentric) or a limited set of tasks (e.g., captioning), failing to capture the full spectrum of MLLM capabilities.
  • Inappropriate tasks: Many tasks can be solved by focusing on isolated frames or by leveraging external knowledge (like movie trivia), rather than requiring a deep understanding of the entire video's narrative or context.

This gap hinders the development of MLLMs that can truly comprehend and reason about complex, lengthy video content.

Methods

The core contribution of this paper is the MLVU (Multi-task Long Video Understanding Benchmark). MLVU is designed to provide a comprehensive and in-depth evaluation of MLLMs' long-video understanding capabilities. Its key features include:

  • Extended Video Lengths: MLVU utilizes videos ranging from 3 minutes to 2 hours. Crucially, tasks can be created for different segments of these videos (e.g., summarizing the first 3 minutes, the first 6 minutes, or the entire duration), allowing for flexible evaluation across various lengths.
  • Diverse Video Genres: The benchmark includes a wide array of video types, such as movies, documentaries, TV series, egocentric videos, life records, sports, tutorials, surveillance footage, cartoons, and game videos. This diversity ensures models are tested in various real-world scenarios.
  • Diversified Evaluation Tasks: MLVU comprises nine distinct tasks designed to assess different facets of long-video understanding. These tasks are categorized into:
  • Holistic LVU: Requires understanding the global information of the entire video (e.g., Topic Reasoning, Anomaly Recognition, Video Summarization).
  • Single-Detail LVU: Focuses on leveraging specific local information within a video (e.g., Needle Question-Answering, Ego Reasoning, Plot Question-Answering, Sub-Scene Captioning).
  • Multi-Detail LVU: Necessitates the joint utilization of multiple plots or details (e.g., Action Order, Action Count). These tasks include both multiple-choice and free-form generation formats.

Datasets

The primary dataset introduced is MLVU, comprising 3,102 questions across 9 tasks. The videos for MLVU are sourced from a collection called ULVC (Universal Long Video Collection), which includes movies (from Movie101, MovieChat), documentaries, game videos (MineDojo), surveillance videos (UCF-Crime), egocentric videos (Ego4D), cartoons, TV series, sports videos, and life records.

Additionally, a derivative dataset called MLVU Time-Ladder was created to specifically study the impact of video length on performance.

Key Insights and Ideas

  • Length is a Major Hurdle: The study unequivocally shows that longer videos pose a significant challenge for current MLLMs. Performance consistently degrades as video length increases, indicating that models struggle to maintain context and recall information over extended durations.
  • Holistic Tasks as a Preliminary Indicator: Close-ended holistic tasks (like Topic Reasoning and Anomaly Recognition) show significant performance variance across models and can serve as a good initial indicator of an MLLM's long-video understanding capabilities.
  • Multi-Detail Tasks are Extremely Difficult: MLLMs, even advanced ones, face "catastrophic degradation" on tasks requiring the understanding of multiple details simultaneously, such as Action Count. This highlights a critical area for future research.
  • RAG Helps with Detail, Not Global Understanding: Retrieval Augmented Generation (RAG) strategies show promise for improving performance on single-detail tasks by helping models retrieve relevant clips. However, RAG offers limited benefits for holistic tasks that require global perception and knowledge aggregation.

Key Results

  • GPT-4o Leads, But Still Struggles: The proprietary model GPT-4o achieved the best performance, with an average score of 54.5% on multiple-choice tasks and 5.87% on generation tasks. However, even GPT-40 struggled significantly, only achieving 42.9% on the Needle Question-Answering task and performing poorly on multi-detail tasks like Action Order and Action Count.
  • Open-Source MLLMs Closing the Gap: Recent open-source long-video MLLMs have made substantial progress, narrowing the gap with proprietary models like GPT-4o. For instance, LLaVA-Onevision trailed GPT-4o by only 2.8% in average multiple-choice performance.
  • Context Length Matters: Increasing the context length for models like MiniGPT4-Video and GPT-40 led to consistent performance improvements, underscoring its importance.
  • Image Understanding Quality is Key: The study suggests that an MLLM's performance on long-video tasks generally aligns with its performance on image understanding benchmarks like MMMU.

Limitations and Future Work

  • Current MLLMs are Not "There Yet": The paper emphasizes that long-video understanding remains a technically challenging problem. All tested models struggled with most tasks, especially those requiring fine-grained information from entire videos.
  • Future Advancements: The authors suggest that future progress will likely require joint optimization of complex factors, including context length, image understanding ability, and the choice of LLM backbone.
  • RAG Limitations: While RAG shows promise for detail retrieval, more effective approaches are needed for global understanding tasks and to improve efficiency.
  • Need for Better Multi-Detail Reasoning: The significant performance drop on multi-detail tasks indicates a critical area for future research and development.

Summary updated

Abstract

The evaluation of Long Video Understanding (LVU) performance poses an important but challenging research problem. Despite previous efforts, the existing video understanding benchmarks are severely constrained by several issues, especially the insufficient lengths of videos, a lack of diversity in video types and evaluation tasks, and the inappropriateness for evaluating LVU performances. To address the above problems, we propose a new benchmark, called MLVU (Multi-task Long Video Understanding Benchmark), for the comprehensive and in-depth evaluation of LVU. MLVU presents the following critical values: 1) The substantial and flexible extension of video lengths, which enables the benchmark to evaluate LVU performance across a wide range of durations. 2) The inclusion of various video genres, e.g., movies, surveillance footage, egocentric videos, cartoons, game videos, etc., which reflects the models' LVU performances in different scenarios. 3) The development of diversified evaluation tasks, which enables a comprehensive examination of MLLMs' key abilities in long-video understanding. The empirical study with 20 latest MLLMs reveals significant room for improvement in today's technique, as all existing methods struggle with most of the evaluation tasks and exhibit severe performance degradation when handling longer videos. Additionally, it suggests that factors such as context length, image-understanding quality, and the choice of LLM backbone can play critical roles in future advancements. We anticipate that MLVU will advance the research of long video understanding by providing a comprehensive and in-depth analysis of MLLMs.
BibTeX
@inproceedings{zhou2025mlvu,
  author = {Junjie Zhou and Yan Shu and Bo Zhao and Boya Wu and Shitao Xiao and Xi Yang and Yongping Xiong and Bo Zhang and Tiejun Huang and Zheng Liu},
  title = {{MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=OaXky6J047},
}
Figure