MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios

Yang Shi, Huanqian Wang, Wulin Xie, Huanyao Zhang, Lijie Zhao, Yi-Fan Zhang, Xinfeng Li, Chaoyou Fu, Zhuoer Wen, Wenting Liu, Zhuoran Zhang, Xinlong Chen, Bohan Zeng, Sihan Yang, Yushuo Guan, Liang Wang, Haoxuan Li, Zhouchen Lin, Yuanxing Zhang, Pengfei Wan, Haotian Wang, Wenjing Yang

TL;DR

While Multimodal Large Language Models (MLLMs) excel at reading text from static images, their performance drops significantly when dealing with text in dynamic videos. To bridge this gap, the authors introduce MME-VideoOCR, a rigorous new benchmark featuring 1,464 videos and 2,000 manually annotated question-answer pairs across 25 distinct tasks. Evaluation of 18 state-of-the-art MLLMs reveals critical blind spots in temporal integration, spatio-temporal reasoning, and a strong vulnerability to language prior biases.

What Problem Does This Paper Address?

Optical Character Recognition (OCR) is a cornerstone of visual understanding. While modern MLLMs can effortlessly extract text from static documents or scenes, video environments introduce a messy cocktail of motion blur, compression artifacts, floating overlays, and temporal variations.

More importantly, text in videos isn't just static information captured in a single frame; it is often distributed sequentially across time. To truly understand a video, a model must not only perceive the text but also reason about how it changes, tracks, or accumulates over multiple frames. Existing video QA benchmarks either treat text as a secondary cue or focus purely on simple perception (e.g., reading a static sign in a video clip). The authors address this lack of comprehensive evaluation by creating a benchmark specifically designed to test text-based perception, understanding, and reasoning in dynamic video scenarios.

Figure 1: An example in MME-VideoOCR. The task requires the MLLM to first recognize the textual information distributed across multiple video frames, and then to perform semantic understanding and reasoning over the extracted text to accurately determine the correct answer. The correct information is marked in blue, while misleading information is marked in red.

Methods

The authors established a meticulous pipeline to construct MME-VideoOCR, focusing on data quality and the elimination of machine-learning biases:

  1. Multi-Source Video Collection & Filtering: Videos were sourced by reconstructing existing video QA datasets, manually scraping public platforms (YouTube, Bilibili, Kuaishou) to include underrepresented formats like vertical videos, and generating custom synthetic videos using Llama 3.1-8B and the Wan text-to-video model.
  2. Two-Stage Manual Annotation: To avoid the hallucinations common in automated LLM annotations, human annotators generated 3–4 QA pairs per video. A second team of experts then filtered these down to the 1–2 highest-quality, most challenging pairs.
  3. Rigorous Debiasing Test: MLLMs often cheat by using internal linguistic knowledge or dataset biases to guess answers without actually "looking" at the video. The authors ran a baseline test presenting the questions to Qwen2.5-VL-7B with no visual input (or a completely black image). Questions where the model could guess the correct answer above random chance (25% for multiple choice) were iteratively revised or replaced.

Datasets

The paper introduces the MME-VideoOCR benchmark:

  • Size: 1,464 videos and 2,000 human-annotated QA pairs.
  • Diversity: Spans 9 major scenario categories (e.g., daily life, sports, business, education) and 44 specific scene types.
  • Structure: Features 10 core task categories broken down into 25 individual tasks, evaluating everything from simple color and text recognition to complex mathematical formula parsing, action tracking, and multi-hop subtitle reasoning.
Figure 2: Example videos and their annotated questions from the MME-VideoOCR benchmark, encompassing 25 tasks across 10 categories. Each task is designed to evaluate models’ capabilities in various aspects such as text recognition, localization, reasoning, and comprehensive video under- standing. The figure displays representative video samples and their corresponding questions.

Key Insights and Ideas

  • The "Language Prior" Trap: One of the most fascinating findings is that MLLMs suffer from a strong language prior bias. When shown visually clear but intentionally misspelled text (e.g., "throuh skin" or "OFF COURS"), models frequently "correct" the text to what makes linguistic sense ("through skin", "OFF COURSE") rather than transcribing what is actually there. This semantic over-reliance is a major hurdle for character-level accuracy.
  • Temporal Integration is Broken: MLLMs perform decently when the answer lies within a single frame or a tight cluster of frames (like standard subtitles). However, when text is scattered across time or requires tracking a trajectory, performance plummets. For instance, models scored terribly on tasks requiring them to piece together scrambled letters appearing sequentially.
  • The Token Compression Penalty: Video models that rely heavily on aggressive token compression or downsampling strategies (e.g., VideoChat-Flash) perform poorly on video OCR. Fine-grained textual details are simply washed away during the token-merging process.
  • Resolution vs. Context Length Trade-off: Increasing input resolution consistently boosts OCR performance across all models. However, increasing the frame count yields diminishing returns: passing 64 frames instead of 32 actually caused a performance dip in some top models, suggesting they struggle with attention allocation over long contexts.

Key Results

The authors evaluated 18 prominent MLLMs, including both proprietary giants and open-source models:

  • The Ceiling: Gemini-2.5 Pro emerged as the top-performing model, but it only achieved an overall accuracy of 73.7%, highlighting the benchmark's difficulty. GPT-4o followed closely at 66.4%.
  • Open-Source Leaders: Among open-source models, Qwen2.5-VL-72B led the pack with 69.0% accuracy, showing a clear scaling effect over its smaller 7B variant (59.1%).
  • The Hardest Tasks:
  • Trajectory Recognition: All top 5 models scored a flat 0.0% on this task, proving that tracking the motion path of an object to spell out a word is currently beyond MLLM capabilities.
  • Scrambled Recognition: Aside from Gemini-2.5 Pro (76%), most models scored below 30% when tasked with reconstructing words from out-of-order characters.
  • Multi-Hop Needle in a Haystack: Models struggled heavily here, with scores ranging between 18% and 28%, confirming their weakness in aggregating scattered temporal information.

Limitations and Future Work

While MME-VideoOCR provides an excellent diagnostic tool, the authors note a few limitations:

  • Data Sparsity: Due to high annotation and evaluation costs, the dataset was capped at 2,000 QA pairs. This means certain niche subcategories have fewer samples, which can cause slight score fluctuations based on a model's sensitivity to specific data points.
  • Evolving Difficulty: Current frontier models already handle the "easy" and "medium" difficulty tiers of this dataset quite well. The authors acknowledge the need to continuously supplement the benchmark with even more complex, long-form, and highly adversarial video samples to keep pace with rapid MLLM advancements.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have achieved considerable accuracy in Optical Character Recognition (OCR) from static images. However, their efficacy in video OCR is significantly diminished due to factors such as motion blur, temporal variations, and visual effects inherent in video content. To provide clearer guidance for training practical MLLMs, we introduce the MME-VideoOCR benchmark, which encompasses a comprehensive range of video OCR application scenarios. MME-VideoOCR features 10 task categories comprising 25 individual tasks and spans 44 diverse scenarios. These tasks extend beyond text recognition to incorporate deeper comprehension and reasoning of textual content within videos. The benchmark consists of 1,464 videos with varying resolutions, aspect ratios, and durations, along with 2,000 meticulously curated, manually annotated question-answer pairs. We evaluate 18 state-of-the-art MLLMs on MME-VideoOCR, revealing that even the best-performing model (Gemini-2.5 Pro) achieves an accuracy of only 73.7%. Fine-grained analysis indicates that while existing MLLMs demonstrate strong performance on tasks where relevant texts are contained within a single or few frames, they exhibit limited capability in effectively handling tasks that demand holistic video comprehension. These limitations are especially evident in scenarios that require spatio-temporal reasoning, cross-frame information integration, or resistance to language prior bias. Our findings also highlight the importance of high-resolution visual input and sufficient temporal coverage for reliable OCR in dynamic video scenarios.
BibTeX
@misc{shi2025mme,
  author = {Yang Shi and Huanqian Wang and Wulin Xie and Huanyao Zhang and Lijie Zhao and Yi-Fan Zhang and Xinfeng Li and Chaoyou Fu and Zhuoer Wen and Wenting Liu and Zhuoran Zhang and Xinlong Chen and Bohan Zeng and Sihan Yang and Yushuo Guan and Zhang Zhang and Liang Wang and Haoxuan Li and Zhouchen Lin and Yuanxing Zhang and Pengfei Wan and Haotian Wang and Wenjing Yang},
  title = {{MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios}},
  year = {2025},
  eprint = {2505.21333},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.21333},
}
Figure