Abstract: Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda, Hansa Meghwani, Jyotika Singh, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track.
2025.
Introduction
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on diverse vision-language benchmarks. However, high scores often mask a fundamental weakness: the "illusion of progress." Models frequently achieve correct answers by identifying localized shortcuts—such as a specific texture or a centered object—rather than performing the holistic, global reasoning required for real-world applications like autonomous driving or medical imaging.
To address this, we introduce the Region Comprehension Index (RCI), a model-based metric designed to quantify whether a benchmark requires global scene understanding or can be solved using isolated local cues. By comparing model performance on full images versus individual image patches, RCI reveals the extent to which datasets reward models for finding localized shortcuts.
Methodology
The RCI framework audits a dataset by comparing a reference model's performance on full images against its performance on individual image patches. The process follows three main steps:
- Patch-Based Evaluation: Each image is partitioned into an $n \times n$ grid (typically $2 \times 2$ or $3 \times 3$).
- Performance Metrics: We calculate Full Image Performance (FIP), the accuracy on the original image, and Maximum Patch Performance (MPP), which aggregates the accuracy achieved using only the single "best" performing patch for each sample.
- RCI Calculation: The index is defined as $RCI_n = 1 - \frac{MPP_n}{FIP}$.
An RCI significantly greater than zero indicates a task requiring strong global reasoning. An RCI near zero suggests balanced reasoning, while a negative RCI reveals that the task can be solved (or even improved upon) using localized visual cues, indicating the presence of dataset shortcuts.
Benchmarking Global vs. Local Reasoning
We applied RCI to 13 widely used multimodal benchmarks using three reference models: InternVL-2.5-1B, Qwen2-VL-2B, and Molmo-1B. Our analysis revealed several critical insights into the current state of multimodal evaluation.
The Shortcut Revelation
Several prominent benchmarks, including BLINK and HallusionBench, exhibited significant negative RCI values (reaching as low as -0.516). This suggests that these datasets often reward models for identifying local fragments, where the full image context may actually be irrelevant or even distracting to the specific question asked.
Global Reasoning Successes
In contrast, benchmarks such as ChartQA and ScienceQA consistently yielded positive RCI values. These tasks require the simultaneous interpretation of multiple elements—such as axes, legends, and data points—forcing models to engage in genuine global reasoning.
Spatial Bias and Model Consistency
Our analysis also uncovered a pervasive "bullseye" effect: the center patch disproportionately contributed to model success across nearly all benchmarks. This spatial bias suggests that dataset creators tend to center critical information, allowing models to ignore the periphery. Notably, RCI trends remained highly consistent ($r > 0.9$) across different model architectures, confirming that RCI measures intrinsic dataset properties rather than model-specific quirks.
Conclusion and Future Work
RCI provides a systematic way to audit multimodal benchmarks for spatial bias and reasoning depth. Human validation confirmed these findings, showing that high-RCI tasks are significantly harder for humans to solve when only shown isolated patches.
While RCI is currently optimized for static images, future work will extend the framework to video and multi-image datasets to evaluate temporal and cross-frame reasoning. Additionally, using RCI as a filtering tool during dataset curation could help eliminate samples that allow for trivial local solutions, pushing the field toward more reliable and robust multimodal AI.
Summary updated