RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks

Amit Agarwal, Hitesh Laxmichand Patel, Subhrakanta Panda, Hansa Meghwani, Jyotika Singh, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth2025 · DOI

Abstract: Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda, Hansa Meghwani, Jyotika Singh, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track.

2025.

Introduction

Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on diverse vision-language benchmarks. However, high scores often mask a fundamental weakness: the "illusion of progress." Models frequently achieve correct answers by identifying localized shortcuts—such as a specific texture or a centered object—rather than performing the holistic, global reasoning required for real-world applications like autonomous driving or medical imaging.

To address this, we introduce the Region Comprehension Index (RCI), a model-based metric designed to quantify whether a benchmark requires global scene understanding or can be solved using isolated local cues. By comparing model performance on full images versus individual image patches, RCI reveals the extent to which datasets reward models for finding localized shortcuts.

Methodology

The RCI framework audits a dataset by comparing a reference model's performance on full images against its performance on individual image patches. The process follows three main steps:

  1. Patch-Based Evaluation: Each image is partitioned into an $n \times n$ grid (typically $2 \times 2$ or $3 \times 3$).
  2. Performance Metrics: We calculate Full Image Performance (FIP), the accuracy on the original image, and Maximum Patch Performance (MPP), which aggregates the accuracy achieved using only the single "best" performing patch for each sample.
  3. RCI Calculation: The index is defined as $RCI_n = 1 - \frac{MPP_n}{FIP}$.
The RCI evaluation pipeline compares model predictions on the whole image versus individual patches to detect reliance on local shortcuts.

An RCI significantly greater than zero indicates a task requiring strong global reasoning. An RCI near zero suggests balanced reasoning, while a negative RCI reveals that the task can be solved (or even improved upon) using localized visual cues, indicating the presence of dataset shortcuts.

Benchmarking Global vs. Local Reasoning

We applied RCI to 13 widely used multimodal benchmarks using three reference models: InternVL-2.5-1B, Qwen2-VL-2B, and Molmo-1B. Our analysis revealed several critical insights into the current state of multimodal evaluation.

The Shortcut Revelation

Several prominent benchmarks, including BLINK and HallusionBench, exhibited significant negative RCI values (reaching as low as -0.516). This suggests that these datasets often reward models for identifying local fragments, where the full image context may actually be irrelevant or even distracting to the specific question asked.

Global Reasoning Successes

In contrast, benchmarks such as ChartQA and ScienceQA consistently yielded positive RCI values. These tasks require the simultaneous interpretation of multiple elements—such as axes, legends, and data points—forcing models to engage in genuine global reasoning.

RCI values across 13 benchmarks and three reference models. Negative values highlight datasets prone to local shortcuts, while positive values indicate a need for global reasoning.

Spatial Bias and Model Consistency

Our analysis also uncovered a pervasive "bullseye" effect: the center patch disproportionately contributed to model success across nearly all benchmarks. This spatial bias suggests that dataset creators tend to center critical information, allowing models to ignore the periphery. Notably, RCI trends remained highly consistent ($r > 0.9$) across different model architectures, confirming that RCI measures intrinsic dataset properties rather than model-specific quirks.

Conclusion and Future Work

RCI provides a systematic way to audit multimodal benchmarks for spatial bias and reasoning depth. Human validation confirmed these findings, showing that high-RCI tasks are significantly harder for humans to solve when only shown isolated patches.

While RCI is currently optimized for static images, future work will extend the framework to video and multi-image datasets to evaluate temporal and cross-frame reasoning. Additionally, using RCI as a filtering tool during dataset curation could help eliminate samples that allow for trivial local solutions, pushing the field toward more reliable and robust multimodal AI.

Summary updated

BibTeX
@misc{agarwal2025rci,
  author = {Amit Agarwal and Hitesh Laxmichand Patel and Subhrakanta Panda and Hansa Meghwani and Jyotika Singh and Karan Dua and Paul Li and Tao Sheng and Sujith Ravi and Dan Roth},
  title = {{RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-industry.10},
}
Figure