PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications

Hitesh Laxmichand Patel, Amit Agarwal, Srikant Panda, Hansa Meghwani, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan RotharXiv 2025 · OpenReview

Abstract

The reliability of Multimodal Large Language Models (MLLMs) in real-world settings is often undermined by sensitivity to irrelevant or distracting visual context, an aspect not captured by existing evaluation metrics. We introduce the Patch Context Robustness Index (PCRI), the first systematic and interpretable score for quantifying MLLM robustness to variations in visual context granularity, measuring performance changes between localized image patches and full-image input. Applying PCRI to 19 state-of-the-art MLLMs across 15 vision-language benchmarks, we find that most leading models remain brittle to background noise, with only a few, such as InternVL2-26B and Qwen2VL-72B, demonstrating consistent robustness across tasks. PCRI analysis also highlights how different model architectures handle and integrate visual context, offering actionable diagnostic insight for both researchers and practitioners. PCRI enables rigorous comparison of context robustness, supporting principled model selection and guiding the development of future architectures and training strategies for robust, real-world deployment.

Introduction: The Challenge of Real-World Visual Context

While MLLMs have made massive strides on academic benchmarks, they often stumble in real-world "enterprise" settings—such as e-commerce search or assistive technologies for the visually impaired. In these scenarios, images are rarely "clean"; they are filled with clutter, occlusions, and irrelevant backgrounds.

Current evaluation protocols usually measure performance on static, full-image contexts, assuming the model is capable of filtering out "noise" to focus on the "signal." However, many models suffer from context brittleness: they get distracted by the background or overwhelmed by excessive visual information. PCRI provides a standardized way to measure this specific type of robustness.

The Patch Context Robustness Index (PCRI)

The core idea behind PCRI is to compare a model's performance on a full image against its performance on localized patches of that same image.

Methodology

  1. Patching: An image is divided into an $n \times n$ grid of non-overlapping patches (e.g., $2 \times 2$ or $3 \times 3$).
  2. Evaluation: The model is asked the same question using the full image ($P_{whole}$) and then using each patch independently.
  3. Max Aggregation: The performance of the best-performing patch ($P_{patch,n}$) is recorded.
  4. The Formula: $PCRI_n = 1 - (P_{patch,n} / P_{whole})$.
The PCRI evaluation pipeline comparing whole-image predictions to patch-based predictions.

Interpreting the Score

  • PCRI $\approx 0$: The model is robust; it performs equally well regardless of context granularity.
  • PCRI $< 0$: The model is "distracted." It actually performs better when shown a small patch than when shown the full image.
  • PCRI $> 0$: The model correctly utilizes global context; it requires the whole image to solve the task.

To ensure reliability, a "Validity Gate" is implemented, only interpreting scores where the model's full-image performance is significantly better than random guessing.

Experimental Scope

The study evaluated 19 state-of-the-art MLLMs (including InternVL2, Qwen2-VL, and Molmo) across 15 vision-language benchmarks. These benchmarks cover four main task types:

  • Image Captioning: MS-COCO.
  • Multiple-Choice QA (MCQ): AI2D, BLINK, MMMU, MMStar, RealWorldQA, ScienceQA.
  • Yes/No Classification: AMBER, HallusionBench, MME, POPE.
  • Visual Question Answering (VQA): ChartQA, GQA, TextVQA, VizWiz.

Key Insights and Results

The "Distraction" Phenomenon

Most benchmarked MLLMs exhibit negative PCRI values, indicating they are frequently distracted by global context. In many cases, models appear to be "guessing" based on background cues in the full image but find the correct answer easily when forced to look at a specific patch. This highlights a fundamental gap between AI "shortcut" learning and human-like contextual reasoning; a human study confirmed that humans almost always perform better with the full image.

Model Scale and Architecture

Larger models, such as InternVL2-26B and Qwen2-VL-72B, emerged as the most robust, maintaining near-zero PCRI across various tasks. This robustness is attributed to more sophisticated attention mechanisms and resolution-adaptive encoding that helps filter out irrelevant tokens.

In contrast, smaller models are notably brittle. For example, InternVL2.5-2B showed extreme negative PCRI (as low as -0.75 on captioning tasks), indicating high susceptibility to background noise.

PCRI scores across models for captioning tasks, highlighting the 'distraction' phenomenon with negative values.

Task and Granularity Sensitivity

Captioning and MCQ tasks showed the highest sensitivity to context noise. Conversely, VQA tasks (like reading a chart) often showed positive PCRI, as they inherently require more global integration. Furthermore, moving from a $2 \times 2$ to a $3 \times 3$ grid generally amplified negative PCRI scores, suggesting that finer-grained local views continue to help models avoid global distractions.

Conclusion

PCRI offers a critical diagnostic tool for researchers and practitioners to evaluate model reliability beyond standard accuracy metrics. While leading large-scale models show improved robustness, the brittleness of smaller models remains a significant challenge for real-world deployment. Future work will expand this framework to sequential data, such as video and audio, and explore multilingual contexts.

Summary updated

Abstract

The reliability of Multimodal Large Language Models (MLLMs) in real-world settings is often undermined by sensitivity to irrelevant or distracting visual context, an aspect not captured by existing evaluation metrics. We introduce the \textbf{Patch Context Robustness Index (PCRI)}, the first systematic and interpretable score for quantifying MLLM robustness to variations in visual context granularity, measuring performance changes between localized image patches and full-image input. Applying PCRI to 19 state-of-the-art MLLMs across 15 vision-language benchmarks, we find that most leading models remain brittle to background noise, with only a few, such as InternVL2-26B and Qwen2VL-72B, demonstrating consistent robustness across tasks. PCRI analysis also highlights how different model architectures handle and integrate visual context, offering actionable diagnostic insight for both researchers and practitioners. PCRI enables rigorous comparison of context robustness, supporting principled model selection and guiding the development of future architectures and training strategies for robust, real-world deployment.
BibTeX
@misc{patel2025pcri,
  author = {Hitesh Laxmichand Patel and Amit Agarwal and Srikant Panda and Hansa Meghwani and Karan Dua and Paul Li and Tao Sheng and Sujith Ravi and Dan Roth},
  title = {{PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications}},
  year = {2025},
  eprint = {2509.23879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23879},
}
Figure