TL;DR
Current multimodal large language models (MLLMs) are often "right for the wrong reasons," producing correct text while looking at completely unrelated parts of an image. This paper introduces VIGOR, a new benchmark that evaluates visual rationale correctness—checking if a model’s internal attention actually aligns with the pixel-level evidence for the concepts it describes. The results are a wake-up call: even the best models struggle to "look at what they say," especially when it comes to complex relations and abstract visual properties.
What Problem Does This Paper Address?
We usually evaluate MLLMs by their output: did the model correctly identify the "traffic light"? However, a model might say "traffic light" simply because it sees a "busy intersection" (a spurious correlation) while its internal attention is actually focused on a tree or a car. This is known as the double correctness problem.
Existing benchmarks focus on output accuracy or post-hoc grounding (linking words to boxes after the fact). They rarely look "under the hood" at the model's internal visual evidence across a wide range of concepts. Furthermore, most grounding datasets are limited to objects, ignoring low-level properties (color, texture) or high-level scene dynamics (actions, relations).
Methods
The authors developed a scalable, automated pipeline to create VIGOR, which represents visual rationales as explicit concepts paired with pixel-level masks.
- Hierarchical Vocabulary: They curated 3,011 unique concepts across four levels: Primitive (color, edges), Intermediate (textures, shapes), Object (parts, whole objects), and Scene (actions, relations).
- Propose-then-Verify Pipeline: To annotate masks at scale without massive human labor, they used a "panel of experts" approach.
- Annotator: A suite of tools (SAM3, CLIPSeg, Grounded-SAM, etc.) generates candidate masks. The authors found that no single tool works for everything; for instance, attention-based tools are better for colors, while SAM is better for objects.
- Verifier: A panel of three MLLMs (Qwen, InternVL, GLM) votes on the best candidate mask. A mask is only accepted if at least two models agree it is accurate and precise.
- Evaluation Protocol: They evaluate MLLMs by extracting their internal attention maps (concept-token attention) and comparing them to the VIGOR ground-truth masks using Average Precision (AP) and Pointing Accuracy.
Datasets
The authors introduce the VIGOR benchmark, consisting of:
- 1,000 images sourced from diverse datasets like COCO, LVIS, and Visual Genome.
- 31,433 pixel-level masks covering the hierarchical concept space.
- 3,011 unique concepts, providing a much denser and more varied annotation set than standard object-detection benchmarks.
Key Insights and Ideas
- The "Spatially Incomplete" Rationale: Models are significantly better at "pointing" (putting their peak attention near the right spot) than at full alignment. Their internal evidence is often fragmented, covering only a small part of the relevant object.
- Layer-wise Non-monotonicity: Rationale correctness follows a "Goldilocks" trend across model layers. Alignment is poor in early layers, peaks in the middle-to-late layers where visual and semantic features integrate, and then drops sharply in the final layers as the model shifts focus to language generation.
- Grounding ≠ Rationale: Interestingly, fine-tuning models specifically for output grounding (e.g., teaching them to generate coordinates) does not significantly improve their internal rationale correctness.
- Hallucination Correlation: There is a strong correlation (Pearson r = 0.70) between rationale correctness and robustness against hallucinations. Models that "look at the right things" are less likely to make things up.
Key Results
The benchmark reveals that current MLLMs have a long way to go:
- Low Absolute Performance: The best-performing model (InternVL3.5-38B) achieved only 31.2% AP and 38.7% Pointing Accuracy overall.
- Concept-Level Failures: Models struggle most with Scene concepts (actions/relations). While they might identify a diagnostic cue, their recall for the full scene evidence is often below 5%.
- Tool Diversity: As shown in the distribution of selected sources, specialized tools are necessary for different levels of abstraction. Grounded-SAM dominates scene concepts, while raw Attention is more useful for primitive visual cues.
Limitations and Future Work
- Attention as a Proxy: The study uses attention maps as a proxy for rationales. While insightful, attention isn't a perfect causal explanation of model behavior.
- Scale: While dense, the benchmark is limited to 1,000 images. Expanding this to specialized domains (like medical imaging) is a logical next step.
- Task Scope: The current evaluation focuses primarily on image captioning. Future work could extend VIGOR to Visual Question Answering (VQA) and more complex reasoning tasks.
Summary updated