Vision-language models (VLMs), which pair an image encoder with a text-generating language model, already fail badly at declining unanswerable questions. On the Unsolvable Problem Detection benchmark, open-source models that score above 80% on ordinary questions detect fewer than 6% of unsolvable ones when given no special prompt. The fixes proposed so far, such as reading uncertainty off the model's first output token or off its middle-layer activations, were all tested on clean, isolated image–question pairs. Real deployments rarely look like that. A question arrives inside retrieved web passages, a long chat history, or a grid of several images. In text-only models, irrelevant retrieved documents drive readers to wrong answers held with 0.98 probability. Nobody has checked whether the internal signals that let a VLM "know it doesn't know" survive that kind of clutter.
This note proposes two things. The first is to measure this question directly with graded-noise versions of existing unanswerability benchmarks. The second is a cheap, training-light abstention score built to be stable under clutter. The score asks how much the model's first-token prediction changes when the image is replaced by a blurred copy, with the noisy context held fixed. The intuition is that distracting context shifts the model's predictions in the same way whether the real or the blurred image is present, so a difference between the two cancels much of the distortion. What remains measures whether the answer actually depends on visual evidence. If it does not, the model is answering from language priors or from the clutter, and it should abstain.
Core Claim
We claim that absolute confidence signals, whether output probabilities, first-token logit probes, or middle-layer probes, degrade sharply as irrelevant context is added. A differential signal, the change in first-token distribution between the real image and a blurred image under identical context, degrades much less. It also transfers to types and amounts of noise that were never seen when its small decision rule was fit.
The closest prior work plays three roles. Focusing Where Vision Matters supplies the method. It defines Visual Information Gain as the drop in answer loss when a blurred image is replaced by the real one, but uses it only offline, to select training data. We turn the same contrast into an inference-time abstention rule and compute it under noisy context. The First to Know and HARMONY are what we stress-test. The first trains a linear probe on first-token logits, raising LLaVA-13B unanswerability detection on VizWiz from 54.92% to 83.45%. The second trains a judge on layer-16 activations plus output probabilities. Neither was evaluated with distractors in the prompt. Improving Zero-shot Reader motivates the work: its distraction-aware method penalizes answers from documents that cannot regenerate the query, but only in text. Our blurred-image contrast is the visual analogue of its "does this evidence explain the query" check.
The Score
Let $z(I, C, q)$ be the first-token logit vector the VLM produces for image $I$, surrounding context $C$ (retrieved passages, extra image tiles, or none), and question $q$. Let $\tilde I$ be a heavily blurred copy of $I$ and $p = \mathrm{softmax}(z)$. The visual-evidence gap is
$$ g(I, C, q) = \mathrm{JS}\big(p(I, C, q)\,\|\,p(\tilde I, C, q)\big), $$
where JS is the Jensen–Shannon divergence. A small $g$ means the model would say roughly the same thing without seeing the image, so its answer is not grounded. We abstain when a logistic combination of $g$, the entropy of $p(I,C,q)$, and the entropy of $p(\tilde I, C, q)$ falls below a threshold. The combination has three parameters and is fit on clean data only. We also compute the same contrast on layer-16 hidden states, following HARMONY's finding that middle layers carry the cleanest visual–text alignment. This lets the study bear directly on the dispute between the two papers over whether hidden knowledge lives in the logits or in middle activations. The cost is one extra prefill and a single token, with no generation.
The approach has a clear failure mode. If context noise interacts with the image, for example a retrieved passage that names an object that is in fact visible, then the shift from noise will not be common to both passes and will not cancel. Measuring how often this happens is part of the study.
Research Program
The first step builds graded-noise variants of three benchmarks. The first is the MM-UPD Bench from Unsolvable Problem Detection for Vision Language Models. It has three settings: the correct option is removed (absent answer), all options are unrelated to the image (incompatible answer set), or the image and question do not match (incompatible visual question). The second is TUBench, with 2,354 near-miss questions over code screenshots, photos, geometry diagrams and tables. The third is VizWiz, which contains real unanswerable photos from blind users.
We apply two families of noise to each item. Textual noise prepends 0, 1, 5, 20 or 50 passages retrieved by text similarity to the question. These are topically related but carry no answer, mimicking multimodal retrieval such as that in RoRA-VLM. Visual noise tiles the target image with 1, 4 or 8 distractor images drawn from LLaVA-CC3M by CLIP distance, reusing the construction of CS-DJ. That paper showed such grids break safety refusals; we ask whether they also break epistemic abstention. Answers to answerable items are unchanged, and a check confirms that the passages contain no answer string.
The second step is an analysis. For LLaVA-1.5-13B and LLaVA-NeXT, the models used in the probing and UPD papers, we trace how each confidence signal separates answerable from unanswerable items as noise grows. We also trace how often the model switches from abstaining to picking an option, the "blind selection" failure. This settles whether clutter pushes VLMs toward confident guessing or toward over-caution, a question left open by the conflict between UPD-style findings and the 96%-abstention finding of Selective "Selective Prediction".
The third step evaluates the gap score against the baselines below under a strict transfer protocol. Every learned component is fit on clean data, or on one noise type at one level, and is tested on all other noise types and levels.
Baselines
The naive reference point is the model as deployed with no abstention mechanism. For multiple-choice benchmarks we add the Option and Instruction prompting settings from the UPD paper, which add "None of the above" and an explicit instruction to use it. On clean MM-UPD, that paper reports that open-source models fall below 6% UPD accuracy in the Base setting. It also reports that instruction tuning brings LLaVA-NeXT-34B dual accuracy to 63.8% on absent-answer, 73.3% on incompatible-answer-set, and 70.2% on incompatible-visual-question items. We rerun these prompts on our models so the comparison is fair.
The strongest existing competitor is the first-token linear probe from The First to Know. It reads the same first token we use, so any gain from our score isolates the effect of the blurred-image contrast rather than of access to the token. Its reported clean result is 83.45% unanswerability-detection accuracy for LLaVA-13B on VizWiz, up from 54.92%. No reported result was found on Lacuna for this probe, for HARMONY, or for any method under added retrieved text or distractor tiles. Those numbers are what this work produces.
We train the probe with the original recipe and give it the same labeled data as our logistic combination. We also give it a larger budget: a variant trained on noise-augmented data. Beating it means higher detection AUROC and dual accuracy on held-out noise levels, at matched false-abstention rate on answerable items. On clean data we would accept parity.
HARMONY is the middle-layer competitor. We reimplement its judge on layer 16 with its own training budget. MaxProb thresholding is the conventional selective-prediction baseline, and we report risk–coverage curves for it.
Two ablations isolate the key ingredient. The first uses only the entropies of the two passes, removing the divergence term $g$. The second replaces the blurred image with a blank image, to check that the effect is visual evidence and not a quirk of input statistics. If the entropy-only variant matches the full score, the contrast adds nothing.
Why not just…?
Why not just retrain the first-token probe on noisy data? This is the real threat. If a probe trained on noise-augmented data generalizes, the differential score is unnecessary. Our transfer protocol settles it. The augmented probe is trained on one noise family, for example text passages at five passages, and tested on tiles, on fifty passages, and on TUBench domains it never saw. Probe transferability is flagged as an open limitation even on clean data. If the augmented probe matches the gap score on held-out noise, we will report that. The contribution then narrows to the benchmark and the degradation analysis, and we would recommend noise augmentation instead.
Why not just filter the context first, as RoRA-VLM's query-oriented token refinement or distraction-aware answer selection do? Filtering targets correctness on answerable questions. It says nothing about whether the image can answer at all. An incompatible-visual-question item stays unanswerable after perfect filtering. We nonetheless run filtered context as an input condition for every method. If filtering alone restores the clean-data probe's performance, that is a strong and useful negative result for our score.
Why not just prompt the model to say "I don't know" or state its confidence? The file's evidence argues against it. The UPD paper finds that Option and Instruction prompts give small gains and sometimes hurt, for example on LLaVA-NeXT. Know What You do Not Know finds that verbalized confidence grows more miscalibrated as images degrade. Self-Knowledge Guided Retrieval Augmentation found direct self-assessment the weakest strategy it tested. We still include both prompts as baselines, because the relevant question is whether they collapse faster under clutter, not whether they are weak on clean data.
Why not just threshold output probability? The distraction-aware reader paper shows a wrong answer held at 0.98 probability under irrelevant documents. HARMONY shows that output probabilities track language fluency. Under clutter, high-confidence errors should multiply, and the MaxProb risk–coverage curve will test this directly.
Decisive Checks
Three results decide the proposal. First, the first-token probe's detection AUROC must fall measurably, by several points, between zero and twenty added passages. If it does not, the premise fails, and the paper becomes a robustness confirmation. Second, on held-out noise types and levels, the gap score must lose clearly less AUROC and dual accuracy than both the clean-trained and the noise-augmented probe. This must hold on MM-UPD, TUBench and VizWiz, with the false-abstention rate on answerable items held fixed so no refusal tax is hidden. Third, the gap score must beat the entropy-only and blank-image ablations. We will also report the fraction of items where noise and image interact, meaning the noise shift differs substantially between the real and blurred passes, as the known limit of the cancellation argument.
References
- Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
- The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?
- HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Models
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models
- TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
- RoRA-VLM: Robust Retrieval-Augmented Vision Language Models
- Distraction is All You Need for Multimodal Large Language Model Jailbreaking
- Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
- Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Models
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models