Abstract
The reliability of Multimodal Large Language Models (MLLMs) in real-world settings is often undermined by sensitivity to irrelevant or distracting visual context, an aspect not captured by existing evaluation metrics. We introduce the Patch Context Robustness Index (PCRI), the first systematic and interpretable score for quantifying MLLM robustness to variations in visual context granularity, measuring performance changes between localized image patches and full-image input. Applying PCRI to 19 state-of-the-art MLLMs across 15 vision-language benchmarks, we find that most leading models remain brittle to background noise, with only a few, such as InternVL2-26B and Qwen2VL-72B, demonstrating consistent robustness across tasks. PCRI analysis also highlights how different model architectures handle and integrate visual context, offering actionable diagnostic insight for both researchers and practitioners. PCRI enables rigorous comparison of context robustness, supporting principled model selection and guiding the development of future architectures and training strategies for robust, real-world deployment.
Introduction: The Challenge of Real-World Visual Context
While MLLMs have made massive strides on academic benchmarks, they often stumble in real-world "enterprise" settings—such as e-commerce search or assistive technologies for the visually impaired. In these scenarios, images are rarely "clean"; they are filled with clutter, occlusions, and irrelevant backgrounds.
Current evaluation protocols usually measure performance on static, full-image contexts, assuming the model is capable of filtering out "noise" to focus on the "signal." However, many models suffer from context brittleness: they get distracted by the background or overwhelmed by excessive visual information. PCRI provides a standardized way to measure this specific type of robustness.
The Patch Context Robustness Index (PCRI)
The core idea behind PCRI is to compare a model's performance on a full image against its performance on localized patches of that same image.
Methodology
- Patching: An image is divided into an $n \times n$ grid of non-overlapping patches (e.g., $2 \times 2$ or $3 \times 3$).
- Evaluation: The model is asked the same question using the full image ($P_{whole}$) and then using each patch independently.
- Max Aggregation: The performance of the best-performing patch ($P_{patch,n}$) is recorded.
- The Formula: $PCRI_n = 1 - (P_{patch,n} / P_{whole})$.
Interpreting the Score
- PCRI $\approx 0$: The model is robust; it performs equally well regardless of context granularity.
- PCRI $< 0$: The model is "distracted." It actually performs better when shown a small patch than when shown the full image.
- PCRI $> 0$: The model correctly utilizes global context; it requires the whole image to solve the task.
To ensure reliability, a "Validity Gate" is implemented, only interpreting scores where the model's full-image performance is significantly better than random guessing.
Experimental Scope
The study evaluated 19 state-of-the-art MLLMs (including InternVL2, Qwen2-VL, and Molmo) across 15 vision-language benchmarks. These benchmarks cover four main task types:
- Image Captioning: MS-COCO.
- Multiple-Choice QA (MCQ): AI2D, BLINK, MMMU, MMStar, RealWorldQA, ScienceQA.
- Yes/No Classification: AMBER, HallusionBench, MME, POPE.
- Visual Question Answering (VQA): ChartQA, GQA, TextVQA, VizWiz.
Key Insights and Results
The "Distraction" Phenomenon
Most benchmarked MLLMs exhibit negative PCRI values, indicating they are frequently distracted by global context. In many cases, models appear to be "guessing" based on background cues in the full image but find the correct answer easily when forced to look at a specific patch. This highlights a fundamental gap between AI "shortcut" learning and human-like contextual reasoning; a human study confirmed that humans almost always perform better with the full image.
Model Scale and Architecture
Larger models, such as InternVL2-26B and Qwen2-VL-72B, emerged as the most robust, maintaining near-zero PCRI across various tasks. This robustness is attributed to more sophisticated attention mechanisms and resolution-adaptive encoding that helps filter out irrelevant tokens.
In contrast, smaller models are notably brittle. For example, InternVL2.5-2B showed extreme negative PCRI (as low as -0.75 on captioning tasks), indicating high susceptibility to background noise.
Task and Granularity Sensitivity
Captioning and MCQ tasks showed the highest sensitivity to context noise. Conversely, VQA tasks (like reading a chart) often showed positive PCRI, as they inherently require more global integration. Furthermore, moving from a $2 \times 2$ to a $3 \times 3$ grid generally amplified negative PCRI scores, suggesting that finer-grained local views continue to help models avoid global distractions.
Conclusion
PCRI offers a critical diagnostic tool for researchers and practitioners to evaluate model reliability beyond standard accuracy metrics. While leading large-scale models show improved robustness, the brittleness of smaller models remains a significant challenge for real-world deployment. Future work will expand this framework to sequential data, such as video and audio, and explore multilingual contexts.
Summary updated