TL;DR
While Multimodal Large Language Models (MLLMs) excel at reading text from static images, their performance drops significantly when dealing with text in dynamic videos. To bridge this gap, the authors introduce MME-VideoOCR, a rigorous new benchmark featuring 1,464 videos and 2,000 manually annotated question-answer pairs across 25 distinct tasks. Evaluation of 18 state-of-the-art MLLMs reveals critical blind spots in temporal integration, spatio-temporal reasoning, and a strong vulnerability to language prior biases.
What Problem Does This Paper Address?
Optical Character Recognition (OCR) is a cornerstone of visual understanding. While modern MLLMs can effortlessly extract text from static documents or scenes, video environments introduce a messy cocktail of motion blur, compression artifacts, floating overlays, and temporal variations.
More importantly, text in videos isn't just static information captured in a single frame; it is often distributed sequentially across time. To truly understand a video, a model must not only perceive the text but also reason about how it changes, tracks, or accumulates over multiple frames. Existing video QA benchmarks either treat text as a secondary cue or focus purely on simple perception (e.g., reading a static sign in a video clip). The authors address this lack of comprehensive evaluation by creating a benchmark specifically designed to test text-based perception, understanding, and reasoning in dynamic video scenarios.
Methods
The authors established a meticulous pipeline to construct MME-VideoOCR, focusing on data quality and the elimination of machine-learning biases:
- Multi-Source Video Collection & Filtering: Videos were sourced by reconstructing existing video QA datasets, manually scraping public platforms (YouTube, Bilibili, Kuaishou) to include underrepresented formats like vertical videos, and generating custom synthetic videos using Llama 3.1-8B and the Wan text-to-video model.
- Two-Stage Manual Annotation: To avoid the hallucinations common in automated LLM annotations, human annotators generated 3–4 QA pairs per video. A second team of experts then filtered these down to the 1–2 highest-quality, most challenging pairs.
- Rigorous Debiasing Test: MLLMs often cheat by using internal linguistic knowledge or dataset biases to guess answers without actually "looking" at the video. The authors ran a baseline test presenting the questions to Qwen2.5-VL-7B with no visual input (or a completely black image). Questions where the model could guess the correct answer above random chance (25% for multiple choice) were iteratively revised or replaced.
Datasets
The paper introduces the MME-VideoOCR benchmark:
- Size: 1,464 videos and 2,000 human-annotated QA pairs.
- Diversity: Spans 9 major scenario categories (e.g., daily life, sports, business, education) and 44 specific scene types.
- Structure: Features 10 core task categories broken down into 25 individual tasks, evaluating everything from simple color and text recognition to complex mathematical formula parsing, action tracking, and multi-hop subtitle reasoning.
Key Insights and Ideas
- The "Language Prior" Trap: One of the most fascinating findings is that MLLMs suffer from a strong language prior bias. When shown visually clear but intentionally misspelled text (e.g., "throuh skin" or "OFF COURS"), models frequently "correct" the text to what makes linguistic sense ("through skin", "OFF COURSE") rather than transcribing what is actually there. This semantic over-reliance is a major hurdle for character-level accuracy.
- Temporal Integration is Broken: MLLMs perform decently when the answer lies within a single frame or a tight cluster of frames (like standard subtitles). However, when text is scattered across time or requires tracking a trajectory, performance plummets. For instance, models scored terribly on tasks requiring them to piece together scrambled letters appearing sequentially.
- The Token Compression Penalty: Video models that rely heavily on aggressive token compression or downsampling strategies (e.g., VideoChat-Flash) perform poorly on video OCR. Fine-grained textual details are simply washed away during the token-merging process.
- Resolution vs. Context Length Trade-off: Increasing input resolution consistently boosts OCR performance across all models. However, increasing the frame count yields diminishing returns: passing 64 frames instead of 32 actually caused a performance dip in some top models, suggesting they struggle with attention allocation over long contexts.
Key Results
The authors evaluated 18 prominent MLLMs, including both proprietary giants and open-source models:
- The Ceiling: Gemini-2.5 Pro emerged as the top-performing model, but it only achieved an overall accuracy of 73.7%, highlighting the benchmark's difficulty. GPT-4o followed closely at 66.4%.
- Open-Source Leaders: Among open-source models, Qwen2.5-VL-72B led the pack with 69.0% accuracy, showing a clear scaling effect over its smaller 7B variant (59.1%).
- The Hardest Tasks:
- Trajectory Recognition: All top 5 models scored a flat 0.0% on this task, proving that tracking the motion path of an object to spell out a word is currently beyond MLLM capabilities.
- Scrambled Recognition: Aside from Gemini-2.5 Pro (76%), most models scored below 30% when tasked with reconstructing words from out-of-order characters.
- Multi-Hop Needle in a Haystack: Models struggled heavily here, with scores ranging between 18% and 28%, confirming their weakness in aggregating scattered temporal information.
Limitations and Future Work
While MME-VideoOCR provides an excellent diagnostic tool, the authors note a few limitations:
- Data Sparsity: Due to high annotation and evaluation costs, the dataset was capped at 2,000 QA pairs. This means certain niche subcategories have fewer samples, which can cause slight score fluctuations based on a model's sensitivity to specific data points.
- Evolving Difficulty: Current frontier models already handle the "easy" and "medium" difficulty tiers of this dataset quite well. The authors acknowledge the need to continuously supplement the benchmark with even more complex, long-form, and highly adversarial video samples to keep pace with rapid MLLM advancements.
Summary updated