TL;DR
This paper introduces MLVU, a new benchmark designed to rigorously evaluate how well multimodal large language models (MLLMs) understand long videos. MLVU addresses the limitations of existing benchmarks by using longer videos, a wider variety of content, and more challenging, diverse tasks, revealing that current MLLMs still struggle significantly with this complex domain.
What Problem Does This Paper Address?
The field of multimodal large language models (MLLMs) is rapidly advancing, with models capable of processing text, images, and even video. However, evaluating their ability to understand long videos (minutes to hours) is a significant challenge. Existing video understanding benchmarks often suffer from:
- Short video lengths: Most benchmarks use videos that are only a few seconds long, which doesn't reflect real-world scenarios or test the models' ability to retain information over extended periods.
- Lack of diversity: Benchmarks tend to focus on a single type of video (e.g., egocentric) or a limited set of tasks (e.g., captioning), failing to capture the full spectrum of MLLM capabilities.
- Inappropriate tasks: Many tasks can be solved by focusing on isolated frames or by leveraging external knowledge (like movie trivia), rather than requiring a deep understanding of the entire video's narrative or context.
This gap hinders the development of MLLMs that can truly comprehend and reason about complex, lengthy video content.
Methods
The core contribution of this paper is the MLVU (Multi-task Long Video Understanding Benchmark). MLVU is designed to provide a comprehensive and in-depth evaluation of MLLMs' long-video understanding capabilities. Its key features include:
- Extended Video Lengths: MLVU utilizes videos ranging from 3 minutes to 2 hours. Crucially, tasks can be created for different segments of these videos (e.g., summarizing the first 3 minutes, the first 6 minutes, or the entire duration), allowing for flexible evaluation across various lengths.
- Diverse Video Genres: The benchmark includes a wide array of video types, such as movies, documentaries, TV series, egocentric videos, life records, sports, tutorials, surveillance footage, cartoons, and game videos. This diversity ensures models are tested in various real-world scenarios.
- Diversified Evaluation Tasks: MLVU comprises nine distinct tasks designed to assess different facets of long-video understanding. These tasks are categorized into:
- Holistic LVU: Requires understanding the global information of the entire video (e.g., Topic Reasoning, Anomaly Recognition, Video Summarization).
- Single-Detail LVU: Focuses on leveraging specific local information within a video (e.g., Needle Question-Answering, Ego Reasoning, Plot Question-Answering, Sub-Scene Captioning).
- Multi-Detail LVU: Necessitates the joint utilization of multiple plots or details (e.g., Action Order, Action Count). These tasks include both multiple-choice and free-form generation formats.
Datasets
The primary dataset introduced is MLVU, comprising 3,102 questions across 9 tasks. The videos for MLVU are sourced from a collection called ULVC (Universal Long Video Collection), which includes movies (from Movie101, MovieChat), documentaries, game videos (MineDojo), surveillance videos (UCF-Crime), egocentric videos (Ego4D), cartoons, TV series, sports videos, and life records.
Additionally, a derivative dataset called MLVU Time-Ladder was created to specifically study the impact of video length on performance.
Key Insights and Ideas
- Length is a Major Hurdle: The study unequivocally shows that longer videos pose a significant challenge for current MLLMs. Performance consistently degrades as video length increases, indicating that models struggle to maintain context and recall information over extended durations.
- Holistic Tasks as a Preliminary Indicator: Close-ended holistic tasks (like Topic Reasoning and Anomaly Recognition) show significant performance variance across models and can serve as a good initial indicator of an MLLM's long-video understanding capabilities.
- Multi-Detail Tasks are Extremely Difficult: MLLMs, even advanced ones, face "catastrophic degradation" on tasks requiring the understanding of multiple details simultaneously, such as Action Count. This highlights a critical area for future research.
- RAG Helps with Detail, Not Global Understanding: Retrieval Augmented Generation (RAG) strategies show promise for improving performance on single-detail tasks by helping models retrieve relevant clips. However, RAG offers limited benefits for holistic tasks that require global perception and knowledge aggregation.
Key Results
- GPT-4o Leads, But Still Struggles: The proprietary model GPT-4o achieved the best performance, with an average score of 54.5% on multiple-choice tasks and 5.87% on generation tasks. However, even GPT-40 struggled significantly, only achieving 42.9% on the Needle Question-Answering task and performing poorly on multi-detail tasks like Action Order and Action Count.
- Open-Source MLLMs Closing the Gap: Recent open-source long-video MLLMs have made substantial progress, narrowing the gap with proprietary models like GPT-4o. For instance, LLaVA-Onevision trailed GPT-4o by only 2.8% in average multiple-choice performance.
- Context Length Matters: Increasing the context length for models like MiniGPT4-Video and GPT-40 led to consistent performance improvements, underscoring its importance.
- Image Understanding Quality is Key: The study suggests that an MLLM's performance on long-video tasks generally aligns with its performance on image understanding benchmarks like MMMU.
Limitations and Future Work
- Current MLLMs are Not "There Yet": The paper emphasizes that long-video understanding remains a technically challenging problem. All tested models struggled with most tasks, especially those requiring fine-grained information from entire videos.
- Future Advancements: The authors suggest that future progress will likely require joint optimization of complex factors, including context length, image understanding ability, and the choice of LLM backbone.
- RAG Limitations: While RAG shows promise for detail retrieval, more effective approaches are needed for global understanding tasks and to improve efficiency.
- Need for Better Multi-Detail Reasoning: The significant performance drop on multi-detail tasks indicates a critical area for future research and development.
Summary updated