MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos

Kejian Zhu, Zhuoran Jin, Hongbang Yuan, Jiachun Li, Shangqing Tu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

TL;DR

The paper introduces MMR-V, a new benchmark designed to evaluate deep, long-range multimodal reasoning in videos rather than shallow perception. By testing models on hidden subtexts and non-adjacent frame dependencies, the authors reveal that even top-tier models like Gemini-2.5-Pro top out at 64.3% accuracy, exposing a massive 21.7% gap compared to human performance.

What Problem Does This Paper Address?

While large language models have made massive leaps in textual reasoning (e.g., OpenAI's o1, DeepSeek-R1), evaluating deep reasoning in the multimodal video domain has lagged behind. Existing video benchmarks predominantly focus on localized perception and understanding tasks—asking models to describe actions or match frames explicitly mentioned in a question.

The authors argue that these tasks fail to test whether a model can truly "think with videos." Real-world video reasoning requires integrating sequential, long-range multi-frame evidence and uncovering hidden information or subtext that isn't explicitly stated in a single "question frame." MMR-V addresses this gap by providing a dataset where shortcuts fail and deep reasoning is mandatory.

Comparison of MMR-V tasks requiring deep reasoning versus previous perception-based video benchmarks.

Methods

Inspired by Kahneman’s Dual Process Theory, the authors categorize video reasoning into two core types:

  1. Implicit Reasoning: Requires extracting underlying subtext, metaphors, emotional shifts, or cultural symbols that operate beneath surface-level visual information.
  2. Explicit Reasoning: Demands rigorous logical deduction across non-adjacent, long-range frames to track causal relationships or counterintuitive events (like magic tricks).

To prevent models from exploiting simple text-matching shortcuts, the authors implemented a clever distractor annotation strategy. They prompted GPT-4o to answer the questions directly; if the model answered incorrectly, that incorrect response was preserved as a highly confusing, model-aligned distractor option.

Datasets

The authors introduced the MMR-V Bench, which consists of:

  • 317 videos curated from YouTube across six major domains: Animation, Film, Philosophy, TV, Life, and Art. The average video length is 277 seconds.
  • 1,257 multiple-choice tasks, with an average of 10 options per task to maximize difficulty.
  • The tasks span 10 categories and 33 subcategories, requiring models to reason over an average of 12 frames, covering roughly 60% of the video duration.
Overview of the six primary task categories within the MMR-V benchmark.

Key Insights and Ideas

  • The Multimodal CoT Illusion: Traditional Chain-of-Thought (CoT) prompting and test-time compute scaling yield surprisingly low gains on MMR-V. Analysis of model outputs revealed that current MLLM reasoning is heavily text-biased; visual analysis accounts for only about 10% of their CoT steps. Models tend to reason textually over the question and options rather than mining the video for evidence.
  • Implicit vs. Explicit Performance: Counterintuitively, models performed better on implicit tasks than explicit ones (by an average margin of +7.9%). In implicit tasks (like theme or metaphor understanding), directors often scatter thematic cues throughout the entire video, giving models abundant chances to guess correctly. Explicit tasks, however, require pinpointing highly localized, specific evidence frames (e.g., a hidden lighter in a magician's hand), which models struggle to locate.
Analysis of CoT content across different stages, showing that stronger models engage in significantly more video analysis of non-question frames.
  • Audio Matters: For models supporting full native multimodality, adding the audio track consistently boosted accuracy (by 1.0% to 1.4%). Audio provides critical context, such as ambient noise levels or spoken puns, that clarifies ambiguous visual data.

Key Results

  • The Human-Model Gap: Humans achieved an average score of 86.0%, whereas the best-performing model, Gemini-2.5-Pro (sampled at 1 fps), achieved only 64.3%.
  • Open-Source vs. Proprietary: Open-source models lagged significantly behind. The top open-source model, Gemma-3-27b-it, scored 42.0%, failing to match proprietary baselines like GPT-4o (52.8%) and GPT-5 (60.6%).
  • Failure Modes: An error analysis of GPT-4o revealed that Lack of Visual Reasoning was the single largest source of failure (47%), underscoring the model's inability to execute long-range, multi-frame evidence localization.
Performance breakdown of various leading models across the different reasoning tasks in MMR-V.

Limitations and Future Work

The authors note several limitations:

  1. High Annotation Cost: Because every QA pair requires meticulous manual verification and cross-referencing with user comments, scaling the dataset is expensive.
  2. Genre Underrepresentation: Complex real-world reasoning genres like mystery, gaming, and puzzle-solving remain underrepresented.
  3. Language Bias: The dataset is predominantly in English, limiting its immediate applicability for evaluating multilingual video reasoning.

Future work will focus on expanding the dataset to include these missing genres and exploring how tool use or specialized multi-frame visual CoT architectures can bridge the human-model performance gap.

Summary updated

Abstract

The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus on understanding tasks, which only require models to match frames mentioned in the question (hereafter referred to as "question frame") and perceive a few adjacent frames. To address this gap, we propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos. The benchmark is characterized by the following features. (1) Long-range, multi-frame reasoning: Models are required to infer and analyze evidence frames that may be far from the question frame. (2) Beyond perception: Questions cannot be answered through direct perception alone but require reasoning over hidden information. (3) Reliability: All tasks are manually annotated, referencing extensive real-world user understanding to align with common perceptions. (4) Confusability: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the best-performing model, Gemini-2.5-pro, achieves only 64.3% accuracy. Additionally, current reasoning enhancement strategies (Chain-of-Thought and scaling test-time compute) bring limited gains. Error analysis indicates that the CoT demanded for multi-modal reasoning differs from it in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.
BibTeX
@inproceedings{zhu2026mmr,
  author = {Kejian Zhu and Zhuoran Jin and Hongbang Yuan and Jiachun Li and Shangqing Tu and Pengfei Cao and Yubo Chen and Kang Liu and Jun Zhao},
  title = {{MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=xk8EqWDPQw},
}
Figure