TL;DR
Vision-Language Models (VLMs) that generate reasoning traces can become confidently wrong by "forgetting" to look at the image as they think. This paper shows this "evidence collapse" is a pervasive problem, and crucially, that its danger depends on the task: it's bad for visual tasks but okay for symbolic ones. This suggests that multimodal monitoring needs to be task-aware, not just a one-size-fits-all approach.
What Problem Does This Paper Address?
Modern AI systems, especially Vision-Language Models (VLMs) that produce step-by-step reasoning (often called "chain-of-thought" or CoT reasoning), are getting remarkably good. However, simply being accurate isn't enough for high-stakes applications. We need to know when they are likely to be wrong. A key challenge is calibration: ensuring the model's confidence in its answer matches its actual accuracy.
This paper focuses on a specific failure mode in VLMs that perform reasoning: evidence collapse. As these models generate a reasoning trace, they might start by attending to the relevant parts of an image, but as the text generation progresses, their attention can drift away from the visual input and rely more on learned text-based priors. This can lead to confident but incorrect answers that are "ungrounded" in the visual evidence. The problem is that standard text-only uncertainty measures (like entropy of the predicted tokens) can't detect this visual disengagement, making it a hidden failure mode. The authors want to understand this phenomenon, quantify it, and develop better monitoring strategies that leverage visual information.
Methods
The core of the paper involves analyzing the internal workings of VLMs, specifically their attention mechanisms, as they generate reasoning.
- Trajectory-level Visual Grounding Measurement: They track how much attention a VLM pays to relevant image regions over the entire reasoning process, not just at the final answer. This is done by sampling attention weights at different stages of generation (start, mid, end of reasoning).
- Evidence Collapse Metric: They define metrics like "total visual attention mass" ($V_t$) and "evidence attention mass" ($Abbox_t$) to quantify how much attention is directed towards the image and specifically towards evidence regions. A decline in these metrics over the reasoning trace indicates evidence collapse.
- Entropy-Vision Interaction Model: To understand the interplay between text-based uncertainty (entropy) and visual grounding, they build a model that predicts correctness based on both entropy and visual engagement. This allows them to identify situations where a model is confident (low entropy) but visually disengaged (low attention).
- Task-Conditional Analysis: They investigate whether the "danger" of confident but visually disengaged predictions varies across different types of tasks. This involves analyzing performance on visual-reference tasks versus symbolic reasoning tasks.
- Vision Veto Policy: They propose a simple monitoring strategy that uses a "vision veto" – if a model is confident but visually disengaged on a task where visual grounding is critical, its prediction can be flagged or rejected.
Datasets
The authors evaluate their methods on three existing benchmarks:
- MathVista: Focuses on mathematical reasoning involving geometric diagrams and algebraic expressions.
- HallusionBench: Designed to induce hallucinations by using adversarially modified visual contexts.
- MMMU_Pro: A comprehensive benchmark covering expert-level STEM questions across physics, chemistry, biology, and engineering.
For their experiments, they use a 300-sample stratified subset from each benchmark, totaling 900 samples.
Key Insights and Ideas
- Evidence Collapse is Pervasive: It's not an isolated incident but a common phenomenon across different VLMs and datasets. Attention to relevant visual evidence significantly drops as reasoning unfolds.
- Visual Disengagement is Task-Dependent: The danger of being confident but visually disengaged is not uniform. It's hazardous for tasks requiring sustained visual reference (like map interpretation) but can be benign or even beneficial for tasks that transition to symbolic reasoning after initial visual extraction.
- Text-Only Uncertainty is Insufficient: While full-response entropy is a good general uncertainty signal, it cannot detect visual disengagement. This highlights the need for multimodal monitoring.
- Trajectory Matters More Than Endpoints: Measuring visual grounding only at the final answer can hide significant evidence collapse that occurs mid-reasoning. The process of reasoning is crucial.
- Naive Fusion of Vision and Text Fails: Simply combining text uncertainty and visual features with a linear model often degrades performance, suggesting a more nuanced, task-aware approach is needed.
Key Results
- Evidence Collapse: Across all models and datasets, visual attention mass ($V_t$) and evidence attention mass ($Abbox_t$) consistently decline from the start to the end of reasoning. For example, $Abbox_t$ decays in 83.5%-100% of samples.
- Full-Response Entropy: This is identified as the strongest text-only baseline for uncertainty monitoring, consistently outperforming answer-span entropy, especially under cross-dataset transfer.
- Task-Conditional Penalty: An interaction model reveals a "confident-but-blind" penalty (higher error risk for confident but visually disengaged predictions) on tasks like MMMU_Pro, but this penalty is absent or inverted on tasks like MathVista.
- Vision Veto Effectiveness: A targeted vision veto strategy, applied conditionally based on task type, can reduce selective risk by up to 1.9 percentage points at 90% coverage, demonstrating the practical utility of task-aware monitoring.
Limitations and Future Work
- Correlation vs. Causation: The study primarily uses attention as a proxy for visual grounding. Establishing causality would require intervention tests (e.g., activation patching).
- Model Scale: The experiments were conducted on models up to 8 billion parameters. Generalization to larger, state-of-the-art RVLMs remains unverified.
- Architectural Diversity: The study focused on two VLM families. Broader architectural coverage might reveal different grounding dynamics.
- Data Size: The analysis used 300 samples per dataset, which limits the statistical power for fine-grained analysis, especially in corner cases.
- Task-Type Inference: For zero-shot deployment, reliable automatic classification of task types (i.e., whether sustained visual grounding is needed) is crucial but not addressed in this work.
Future work suggested includes:
- Moving from correlation to intervention to prove causality.
- Scaling experiments to larger models and exploring more architectures.
- Developing methods for automatic task-type inference to enable zero-shot conditional monitoring.
Summary updated