Vision-language models, meaning language models that also read images, routinely answer questions that the image cannot support. They describe objects that were masked out, count things that are off-screen, and click the button that looks right on a user interface that is actually ambiguous. Several independent findings suggest that these models often detect the problem internally and still fail to act on it. Telling a model why a question is unanswerable makes it abstain easily (UNK-VQA). Frontier models can write unanswerable visual questions but cannot recognize the same questions when asked to answer them (CertainlyUncertain). Hallucinated and faithful generations separate along a straight line in the model's middle layers (SHARP). Text-only agents recognize that a target is a production system and attack it anyway (A New Framework for Cybersecurity Refusals in AI Agents). If this detection-without-action gap holds, then hallucination on unanswerable inputs is partly a failure of the decision to refuse, and only partly a failure of perception.
We propose to measure that gap directly in open vision-language models, and then to close it with a cheap mechanism. A small linear probe reads the model's hidden state before it generates anything and estimates whether the visual evidence is sufficient. Only when the probe fires does the agent enter a short, structured evidence check, after which it chooses among three first-class actions: answer, refuse, or ask the user for clarification. The two-sentence version is this: the model already knows, inside its activations, when the image can't answer the question, so we read that signal and use it to trigger deliberation and a refusal action only where needed. This should cut hallucinated answers and unsafe clicks on ambiguous interfaces without the helpfulness loss that fine-tuning for honesty has produced.
Core Claim
We make two claims, and the first is a measurement. On unanswerable visual inputs, a linear probe on intermediate hidden states separates answerable from unanswerable cases far better than the model's own verbal behavior does. The gap is widest exactly where models hallucinate most confidently. That means the "Yes"-biased, high-co-occurrence cases in POPE, and the Gaussian-noise condition in MoHoBench, where refusal rates fall even though evidence has gotten worse. The second claim is about intervention. Using that probe as a gate for selective deliberation plus an explicit refusal-or-clarify action beats three alternatives at a fixed false-refusal rate on answerable inputs: always-on deliberation, direct probe-triggered refusal, and honesty fine-tuning. We test this on static benchmarks and on the single-step computer-use setting of ADeptS-Bench.
What is new is neither the probe nor the refusal tool alone. It is the demonstration that perception-level evidence insufficiency is linearly readable before decoding, and that this signal can drive a deliberative agent action rather than a text string. The first half of that is a diagnostic claim no prior work in the file tests. It can also fail cleanly: if probe accuracy is no better than verbal refusal, the "knows but doesn't say" story is wrong for vision-language models.
Where This Sits
Four papers build static benchmarks for multimodal abstention and motivate the gap: MoHoBench, TUBench, UNK-VQA, and CertainlyUncertain.
- MoHoBench builds 12,000 unanswerable questions and finds an average refusal rate of 21.3% across 28 models.
- TUBench builds near-miss unanswerable questions over code, images, geometry, and tables, and finds 17 of 21 open models below 30% F1 on unanswerables.
- UNK-VQA perturbs VQAv2 and tests a classifier head and an entropy threshold.
- CertainlyUncertain fine-tunes on 178K inpainted contrastive pairs.
All four treat abstention as an output string produced by the answering pass. None asks whether the relevant signal already exists in the hidden state before the model commits to an answer.
Two further lines supply our mechanism. SHARP shows hallucination is linearly separable at layers 10–15 of LLaVA models. It uses that separability to suppress prior-driven tokens and push the model toward an answer, which cannot work when the pixels simply do not contain the answer. We reuse its finding of linear separability, but steer toward a decision instead of away from tokens. Adaptive Safety Probing probes layer 8 of a robot control model to trigger verification cheaply, cutting latency by 69.5% versus always-on checking. That supplies the precedent for probe-triggered, resource-light gating, though its probe predicts grasp coordinates in a single robotics domain rather than whether evidence is sufficient.
GRAIT constrains the design. In text-only models it finds that the gradients for learning facts and for learning to refuse are nearly orthogonal, with an inner product of about 0.008. This suggests an evidence-sufficiency direction can be read without disturbing answering ability, but it has never been checked with visual inputs.
The agent side comes from MOSAIC. It gives text agents a plan–check–act/refuse loop, a first-class refuse_unsafe_task tool, and a learned gate that runs safety reasoning only when needed. It reports up to 50% less harmful behavior. MOSAIC never sees an image, and its gate operates on text. ADeptS-Bench is the closest bridge: computer-use agents with a status(unsafe) tool and a disambiguation stream for ambiguous interfaces. It shows that open Qwen-VL models largely ignore the tool, while Claude 4.7 Opus over-refuses. It does not look inside the models, and offers no gating mechanism.
Our proposal differs from ADeptS-Bench in adding the internal diagnosis and the gate. It differs from MOSAIC in making the gate perceptual and pre-generation.
Research Program
We work with open models whose activations we can read. LLaVA-1.5-7B gives direct comparability with CertainlyUncertain and SHARP. A Qwen-VL model gives comparability with ADeptS-Bench's open-source results.
Diagnostic stage. For each input, we extract the residual-stream state at the final prompt token, across all layers. We train a logistic probe per layer to predict answerability. Training uses the UNK-VQA and CertainlyUncertain splits, which have controlled perturbations such as object masking, image replacement, and inpainted erasure. We then report held-out transfer, without retraining, to TUBench's four sub-datasets and to MoHoBench's four categories.
The key quantity is the detection–action gap. At matched false-refusal on answerable items, it is the probe's recall on unanswerables minus the model's verbal refusal recall. We compute it overall and broken down by three conditions: MoHoBench's noise and contrast conditions, POPE's popular versus adversarial negatives, and the four ADeptS-Bench threat levels. The noise condition is especially telling. MoHoBench's finding that Gaussian noise lowers refusal predicts one of two outcomes. Either the probe signal rises under noise, which would mean decision failure, or it falls, which would mean perception failure. That result alone says which part of the pipeline to fix.
Intervention stage. We use the probe as a gate. With $h_\ell(x)$ the hidden state at layer $\ell$ for image-plus-prompt $x$, $w$ and $b$ the probe's weights and bias, and $\sigma$ the logistic function, the agent enters deliberation when
$$ \sigma\!\left(w^\top h_\ell(x) + b\right) > \tau , $$
where $\tau$ is chosen on a held-out split to hit a target false-refusal rate on answerable inputs, for example 5%.
Deliberation is a short, fixed-format turn. The model states what visual evidence the question or action requires, then states whether that evidence is visible, and then emits one of three actions. answer returns to normal decoding. refuse_insufficient_evidence records a reason. request_clarification names the ambiguous referent. Ungated inputs skip deliberation entirely, so cost on routine inputs is one dot product.
The probe is the only trained component. The base model is untouched, which is the design's main defense against a helpfulness tax. As a secondary arm, we also apply the gate on top of a CertainlyUncertain-fine-tuned model, to test whether the two are complementary.
Baselines
The naive reference is the unmodified model with a plain instruction that it may say "I don't know" or call the refusal and clarification tools. This is what MoHoBench effectively measures (21.3% average refusal). On ADeptS-Bench it corresponds to the reported open-source behavior, with Qwen attack success rates often above 75%. It is run with the same tool definitions and prompt template as every other arm.
The strongest existing competitor is honesty fine-tuning on CertainlyUncertain. It is the only method in the file that claims large abstention gains with no tax on answerable questions: LLaVA-1.5-7B's unanswerable F1 rose from about 33% to over 85%, while VQAv2 and VizWiz held. We reproduce it on LLaVA-1.5-7B with the released data and the paper's recipe, and evaluate it on the same held-out benchmarks. Its training data is the same data the probe sees, so neither method gets extra supervision.
Beating it means two things. First, matching or exceeding its unanswerable F1 on held-out TUBench and MoHoBench, not on its own distribution. Second, doing so at equal or lower loss on answerable items. The answerable-item measures are TUBench's overall accuracy (OAAC), the MME perception score (SHARP reports 510.0 for vanilla LLaVA-v1.5), and POPE accuracy (RITUAL reports 84.13% baseline and 88.87% with its method on LLaVA-1.5).
We also run MoHoBench-style preference alignment (DPO). That paper reports refusal rising from about 28% to over 90% at the cost of a "slight performance dip" on MMMU; no exact MMMU figure is reported. Including it lets us quantify that dip ourselves.
For answerability classification, the reference points come from UNK-VQA. GPT-4V reaches 58% binary answerability accuracy zero-shot, and open models reach 40–50%. UNK-VQA's own classifier head and entropy threshold are run as external-detector baselines on identical splits.
Four ablations isolate the proposal's ingredients. A random gate firing at the same rate as the probe tests whether the probe's choice of inputs matters, rather than just the added deliberation. Always-on deliberation, the MOSAIC-without-gate variant, tests whether gating saves the refusal tax. Probe-triggered direct refusal with no deliberation tests whether the deliberative step adds anything beyond the classifier. An entropy-based gate at matched firing rate tests whether the hidden state carries more than the output distribution.
All arms share one tuning budget: $\tau$ is set by the same held-out rule, and prompt tuning is limited to one shared template.
Why not just…?
Why not just fine-tune for honesty, as CertainlyUncertain does? This is the real threat. CertainlyUncertain reports no refusal tax on VQAv2 and VizWiz. That conflicts with MoHoBench and UNK-VQA, which both report taxes, and the file does not resolve the conflict. Our reproduction on held-out benchmarks will settle it for our models.
If fine-tuning matches the gate on held-out F1 and answerable accuracy, the intervention claim fails. The diagnostic claim would still stand, and fine-tuning would be the recommended fix. Two things still favor the gate in that case. It applies to models whose weights cannot be tuned. And it produces an action with a stated reason, whereas CertainlyUncertain's model emits a generic "I don't know." We will also test gate plus fine-tuning. If the combination beats both parts, the two are complementary rather than competing.
Why not just threshold confidence or entropy? UNK-VQA already tested an entropy-based abstainer and found zero-shot performance near chance. POPE shows "Yes" rates near 99%, meaning hallucinations come with low entropy. We include entropy gating at a matched firing rate, so this failure is measured on our models rather than assumed. The hidden-state probe must beat it for the proposal to hold.
Why not just prompt the model to check the image first, every time? That is our always-on deliberation ablation. MOSAIC's motivation for selective gating was token overhead on routine tasks, and Claude 4.7 Opus's over-refusal on benign interfaces in ADeptS-Bench shows that heightened caution everywhere costs utility. If always-on deliberation matches the gated version on both unanswerable recall and answerable accuracy, the gate is unnecessary except for cost. We will then report the savings honestly as a latency result rather than an accuracy result.
Why not just use the probe as a classifier and refuse directly? This is close to UNK-VQA's classifier head. It is the ablation most likely to embarrass the design. If direct refusal matches the deliberative version, then deliberation adds nothing, and the clarification action is the only remaining difference. That difference is measurable on the ADeptS-Bench disambiguation stream, where refusing and asking for clarification are scored differently.
Proposed Experiments
The static evaluation uses held-out TUBench and MoHoBench, with UNK-VQA and CertainlyUncertain test splits for in-distribution checks. We report TUBench's answerability accuracy (2ACC), unanswerable F1, and OAAC, plus MoHoBench refusal rate and refusal rationality. Utility is tracked on MME perception, POPE (random, popular, and adversarial splits), and VQAv2 answerable accuracy.
The agentic evaluation uses the ADeptS-Bench disambiguation stream (744 tasks) and the paired safety stream, reported as task success and attack success rate against its stated frontier: no model exceeded 80% task success with attack success below 30%. We stratify by threat level, expecting gains mainly at the ambiguity-driven levels and little at level-four consequence reasoning. The reason is that the probe reads evidence sufficiency, not downstream consequences. We report that limitation rather than overclaim.
Three decisive checks determine the outcome. The first is whether the detection–action gap is large and positive on held-out benchmarks. If the probe cannot beat verbal refusal at matched false refusal, the premise fails. The second is whether the probe gate beats the random gate at equal firing rate, and beats the entropy gate, on unanswerable F1. The third is whether the gated agent matches CertainlyUncertain fine-tuning on held-out unanswerable F1 while losing less on MME and OAAC. The noise-versus-contrast breakdown will say whether the failure is in perception or in the decision, which determines whether the gate or better visual grounding is the right fix.
References
- UNK-VQA: A Dataset and A Probe into Multi-modal Large Models' Abstention Ability
- CertainlyUncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- A New Framework for Cybersecurity Refusals in AI Agents
- Evaluating Object Hallucination in Large Vision-Language Models
- MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions
- ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
- TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
- Adaptive Safety Probing for Resource-Efficient Vision-Language-Action Models
- GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigation
- Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use