TL;DR
The paper introduces MMR-V, a new benchmark designed to evaluate deep, long-range multimodal reasoning in videos rather than shallow perception. By testing models on hidden subtexts and non-adjacent frame dependencies, the authors reveal that even top-tier models like Gemini-2.5-Pro top out at 64.3% accuracy, exposing a massive 21.7% gap compared to human performance.
What Problem Does This Paper Address?
While large language models have made massive leaps in textual reasoning (e.g., OpenAI's o1, DeepSeek-R1), evaluating deep reasoning in the multimodal video domain has lagged behind. Existing video benchmarks predominantly focus on localized perception and understanding tasks—asking models to describe actions or match frames explicitly mentioned in a question.
The authors argue that these tasks fail to test whether a model can truly "think with videos." Real-world video reasoning requires integrating sequential, long-range multi-frame evidence and uncovering hidden information or subtext that isn't explicitly stated in a single "question frame." MMR-V addresses this gap by providing a dataset where shortcuts fail and deep reasoning is mandatory.
Methods
Inspired by Kahneman’s Dual Process Theory, the authors categorize video reasoning into two core types:
- Implicit Reasoning: Requires extracting underlying subtext, metaphors, emotional shifts, or cultural symbols that operate beneath surface-level visual information.
- Explicit Reasoning: Demands rigorous logical deduction across non-adjacent, long-range frames to track causal relationships or counterintuitive events (like magic tricks).
To prevent models from exploiting simple text-matching shortcuts, the authors implemented a clever distractor annotation strategy. They prompted GPT-4o to answer the questions directly; if the model answered incorrectly, that incorrect response was preserved as a highly confusing, model-aligned distractor option.
Datasets
The authors introduced the MMR-V Bench, which consists of:
- 317 videos curated from YouTube across six major domains: Animation, Film, Philosophy, TV, Life, and Art. The average video length is 277 seconds.
- 1,257 multiple-choice tasks, with an average of 10 options per task to maximize difficulty.
- The tasks span 10 categories and 33 subcategories, requiring models to reason over an average of 12 frames, covering roughly 60% of the video duration.
Key Insights and Ideas
- The Multimodal CoT Illusion: Traditional Chain-of-Thought (CoT) prompting and test-time compute scaling yield surprisingly low gains on MMR-V. Analysis of model outputs revealed that current MLLM reasoning is heavily text-biased; visual analysis accounts for only about 10% of their CoT steps. Models tend to reason textually over the question and options rather than mining the video for evidence.
- Implicit vs. Explicit Performance: Counterintuitively, models performed better on implicit tasks than explicit ones (by an average margin of +7.9%). In implicit tasks (like theme or metaphor understanding), directors often scatter thematic cues throughout the entire video, giving models abundant chances to guess correctly. Explicit tasks, however, require pinpointing highly localized, specific evidence frames (e.g., a hidden lighter in a magician's hand), which models struggle to locate.
- Audio Matters: For models supporting full native multimodality, adding the audio track consistently boosted accuracy (by 1.0% to 1.4%). Audio provides critical context, such as ambient noise levels or spoken puns, that clarifies ambiguous visual data.
Key Results
- The Human-Model Gap: Humans achieved an average score of 86.0%, whereas the best-performing model, Gemini-2.5-Pro (sampled at 1 fps), achieved only 64.3%.
- Open-Source vs. Proprietary: Open-source models lagged significantly behind. The top open-source model, Gemma-3-27b-it, scored 42.0%, failing to match proprietary baselines like GPT-4o (52.8%) and GPT-5 (60.6%).
- Failure Modes: An error analysis of GPT-4o revealed that Lack of Visual Reasoning was the single largest source of failure (47%), underscoring the model's inability to execute long-range, multi-frame evidence localization.
Limitations and Future Work
The authors note several limitations:
- High Annotation Cost: Because every QA pair requires meticulous manual verification and cross-referencing with user comments, scaling the dataset is expensive.
- Genre Underrepresentation: Complex real-world reasoning genres like mystery, gaming, and puzzle-solving remain underrepresented.
- Language Bias: The dataset is predominantly in English, limiting its immediate applicability for evaluating multilingual video reasoning.
Future work will focus on expanding the dataset to include these missing genres and exploring how tool use or specialized multi-frame visual CoT architectures can bridge the human-model performance gap.
Summary updated