Telling "I know this" apart from "the evidence says this" in multimodal models

When a vision-language model is handed an image and a retrieved document and told to answer only from them, it often answers anyway, even when neither source contains the answer or the two sources disagree. Most work treats this as a failure to abstain. We think it is something more specific. The model has three sources of belief: what it sees, what it reads, and what it memorized in training. Its sense of confidence is set mostly by the third, memory. On today's benchmarks, memory is also the source the experimenter cannot control, because benchmark images and facts leak into training data. As a result, we cannot currently tell whether a model abstains badly because it cannot judge whether the evidence is sufficient, or because memory drowns that judgment out.

This proposal builds a procedurally generated multimodal test in which every item's answerability from each source is fixed by construction. Fictional entities appear in generated images and fictional documents, so the memorized prior is known to be absent. These are paired with real-entity twins whose memorization we measure directly. On this test we ask one question: does the model hold an internal signal of evidence sufficiency, separate from its output confidence, that survives when a strong memorized answer is present? If such a signal exists, a cheap linear readout of it should drive abstention better than output confidence, prompting, or preference fine-tuning, without making the model refuse questions it should answer. If no such signal exists, the testbed alone still shows how much of measured abstention failure is a memorization artifact.

Core Claim

Our hypothesis is that abstention failure in multimodal retrieval-augmented question answering is concentrated in one cell. In that cell the model has a memorized answer, but the image and document either lack it or contradict each other. We further hypothesize that mid-to-late-layer hidden states encode evidence sufficiency in that cell even when the output does not reflect it. Two predictions follow. First, a probe trained only on fictional-entity items, where memory is silent, should transfer to real-entity items, where memory is loud. Second, in exactly the cell where memory is present and evidence is absent, gating the answer on that probe should beat gating on token-level confidence. Output confidence should fail there because it tracks the memorized answer. If output confidence does as well as the probe, we will say so, and the contribution reduces to the controlled testbed and the measurement result.

How this differs from the closest work

Four papers motivate the gap, and none of them combines missing-or-conflicting evidence with visual input and a controlled prior.

FaithEval builds unanswerable, inconsistent, and counterfactual contexts for text-only models. It shows memorized facts intruding: Phi-3-medium falls from 75.8% on original contexts to 7.4% when it must admit the context lacks the answer. It has no images and does not control the prior.

MMKC-Bench brings knowledge conflict into the multimodal setting. It finds that models detect conflicts 75–79% of the time yet still answer from memory, and that larger models are more stubborn. It scores only a forced choice between memory and context, never abstention, and its items come from Wikipedia and Google, so they may be contaminated.

MoHoBench measures refusal on unanswerable visual questions, with an average refusal rate of only 21.3% across 28 models. It has no retrieved text and no conflict.

AbstentionBench shows that reasoning models abstain 24% worse than instruction-tuned models and often state the missing information in their thinking traces before answering anyway. It is text-only.

Two lines of work supply the method. Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning shows that pairwise conflicts (vision against prior, vision against text, prior against text) are linearly separable in layers 15–22 of 7B models, with 93.2–98.8% AUC. Steering with those probes cuts high-confidence errors by up to 55%. It is the closest prior work. We differ in three ways:

  • Its probes detect conflict and steer the model toward one source. Ours read sufficiency and trigger abstention when no valid source exists.
  • Its data are adapted static benchmarks. Ours control whether the model has a memorized answer.
  • Its pairwise setting leaves three-way cases unaddressed, which is exactly where we evaluate.

Separately, the two-pass evaluation from Can LLMs Refuse Questions They Do Not Know? compares a normal pass with a forced-answer pass and summarizes the result as the Refusal Index. That index tracks expensive calibration metrics with 85.8% correlation and resists prompt gaming. We reuse its forced-answer pass to measure each item's memorized prior and use the index as our headline metric.

The contamination literature constrains the design. VLB reports up to 84.46% overlap between static multimodal benchmarks and pretraining data. It generates fresh variants by editing images (add, remove, expand) but deliberately keeps them answerable. MM-DETECT shows that text-only leakage inflates multimodal scores. INVITE generates fresh invalid questions by distorting knowledge-graph facts, but only in text. We borrow VLB's editing operations and INVITE's distortion logic, aimed at the opposite goal: making items unanswerable under control.

Research Program

Generator. We build a generator over a small typed world of entities with visual attributes, relations, and textual facts. Each item has three parts, each varied independently:

  • An image that shows the queried fact, omits it, or shows a contradicting value. Images are produced by generation, or by inpainting edits in VLB's style.
  • A document that states the fact, omits it, or contradicts it. Documents are synthetic news-style passages, following NeoQA's practice of fictional events. NeoQA also shows that high word overlap between query and document triggers answering, so overlap is a nuisance factor we vary deliberately.
  • A memorized prior for the entity: absent (fictional entities) or present (real entities).

For real entities, the prior is verified per model and per item with the forced-answer pass. Contamination of the real images is checked with MM-DETECT slot guessing.

Labels. The instruction is to answer from the provided image and document only. The gold label follows mechanically from the cell:

  • answer when at least one source supplies the fact and none contradicts it;
  • abstain when neither source supplies it, even if the model knows it from memory;
  • abstain when the image and document disagree, following FaithEval's convention for inconsistent contexts.

Because items are regenerated per evaluation round, the test cannot be memorized.

Probe and gate. We extract hidden states at the final prompt token across layers and train a logistic probe on the gold answer/abstain label, using fictional-entity items only. The gate abstains when the probe's abstain probability exceeds a threshold. That threshold is chosen on a held-out fictional validation set, never on real-entity items. For reasoning models, we also read the probe at intermediate points in the reasoning trace. This tests whether sufficiency is encoded before the late switch that What Color Is It? documents, where models identify the true ink color early in reasoning and then answer with the word printed in the image.

Models. We use open models named in the file: Qwen2.5-VL-7B, InternVL3.5-14B, and LLaVA-1.5, plus one reasoning model (GLM4v-Thinking or Qwen3-VL). White-box access is required for the probe.

Baselines

The naive reference point is the model with a plain instruction to answer from the given image and document. MoHoBench's 21.3% average refusal rate on unanswerable visual questions is the expected scale of failure. A cautious-prompt variant adds "say 'I don't know' if the evidence is missing or contradictory." We report it to test the claim from Can LLMs Refuse Questions They Do Not Know? that such prompts move the refusal threshold without improving the Refusal Index. A Chain-of-Note-style prompt asks the model to write a relevance note for each source before answering, following Chain-of-Note, which trains such notes and reports a +10.5 rejection-rate gain on RealTimeQA in text.

The strongest training-free competitor is output-confidence gating. The model abstains when the probability or entropy of its answer tokens crosses a threshold, with the threshold tuned on the same fictional validation set as our probe. No reported result was found on Lacuna for confidence gating on multimodal retrieval-augmented abstention, so we establish it ourselves. This baseline has the same inputs, labels, and tuning budget as the probe. A null result against it is the main way the core claim could fail.

The strongest existing white-box competitor is the conflict probe from Diagnosing Knowledge Conflict, reimplemented on our models at its reported layers. It is trained on conflict-versus-no-conflict labels instead of sufficiency labels, with the same probe capacity and training data size. Comparing the two isolates whether sufficiency is a different signal from conflict. A conflict probe cannot, in principle, flag the "both sources silent" cell.

The strongest training-based competitor is DPO on unanswerable items, as in MoHoBench, where refusal rose from about 28% to over 90% at a cost to MMMU performance. The paper gives no MMMU number, so no reported result was found on Lacuna for the size of that cost. We train DPO on the same fictional training split the probe sees. For visual-only unanswerability we include CLIP-UP, which feeds image-text correlation vectors from a CLIP-style model into the VLM and reports dual-accuracy gains of up to 8.58% over LoRA fine-tuning on MM-UPD. We run CLIP-UP on MM-UPD to confirm our implementation reproduces that gain before using it on our test.

Two ablations isolate the key ingredient. The first trains the probe on real-entity items only. If a probe trained on real entities transfers to fictional ones but not the reverse, the signal is entangled with memory. The second removes the image and keeps only the document. This tells us whether vision contributes to the sufficiency representation or whether we have rediscovered a text-only effect.

Beating a baseline means the following. On real-entity items in the "memory present, evidence absent or contradictory" cells, the probe gate must achieve a higher Refusal Index and higher dual accuracy than output-confidence gating and the conflict probe. Dual accuracy requires both correct answers on answerable items and correct abstention on unanswerable ones. The probe gate must also keep over-abstention on answerable items, and the drop on MMMU, at or below the plain-prompt level. DPO is expected to reach high refusal, so it is beaten only if the gate matches its refusal with a smaller loss on answerable items.

Why not just…?

Why not just prompt the model to say "I don't know"? The two-pass analysis in Can LLMs Refuse Questions They Do Not Know? shows that cautious prompting discards answerable questions without changing the Refusal Index. FaithEval reports that prompting for conflicts makes models invent conflicts in clean contexts. Our cautious-prompt baseline tests whether either pattern carries over to images. We expect higher refusal, an unchanged Refusal Index, and more over-abstention on answerable cells.

Why not just threshold the model's own answer confidence? This is the real threat. If memory does not inflate confidence in the decisive cell, confidence gating will match the probe, and our mechanistic claim fails. Indirect evidence that it will not match: MMKC-Bench models detect conflict yet answer from memory, and AbstentionBench reasoning traces name the gap and then answer confidently. Both suggest the sufficiency judgment and the emitted answer come apart. The cell-by-cell comparison settles the question directly, and we will report a null result as such.

Why not just fine-tune for refusal with DPO? MoHoBench shows DPO works for refusal but costs helpfulness. DyVal and VLB show that static training sets give brittle behavior once complexity shifts. Because our generator is procedural, we can train DPO on one world configuration and test on a held-out one with new relations and edit types. That measures generalization, not just in-distribution refusal. If DPO generalizes and keeps answerable accuracy, it wins, and the probe becomes a diagnostic tool, not a fix.

Why not just reuse the existing conflict probe? A conflict probe fires when two sources disagree. The most common failure, missing evidence while memory supplies an answer, involves no disagreement between image and document. The paired comparison above tests this. We expect the conflict probe to do well on contradiction cells and near chance on silent-evidence cells.

Why not just evaluate on MMKC-Bench, MoHoBench, and FaithEval as they are? None of them varies the memorized prior independently of the evidence. With up to 84.46% benchmark overlap with pretraining data (VLB), a model's memory may silently answer "unanswerable" items. We still use these benchmarks for transfer. The probe trained on fictional items is evaluated zero-shot on MoHoBench, on MM-UPD, and on FaithEval's text-only unanswerable split with the image removed. This shows whether results on our test predict results on existing benchmarks.

Proposed Experiments

The first experiment is purely measurement. For each model, we compare the abstention rate in silent-evidence and contradiction cells between fictional and real-entity twins. The prediction is that abstention collapses specifically when the forced-answer pass shows a memorized answer. We stratify real-entity items by MM-DETECT contamination score to test whether more strongly memorized items show worse abstention. A flat result across fictional and real arms would falsify the claim that memory drives the failure.

The second experiment is the decisive check. We compare the probe gate, output-confidence gating, and the conflict probe on real-entity items in the memory-present, evidence-absent cells, using the Refusal Index and dual accuracy, with fictional-trained transfer as the only training regime. We also report the layer profile, expecting the signal in the mid-to-late band that Diagnosing Knowledge Conflict identified.

The third experiment tests reasoning models. We read the probe at successive points in the reasoning trace and stop with an abstention when it fires. We measure whether this recovers the abstention loss that AbstentionBench attributes to long reasoning, and whether it raises accuracy on What Color Is It?, where GPT-4o falls to 9% on Type 4 items. That benchmark is a forced choice, not abstention, so we treat it as a transfer test of whether the early-trace signal tracks the visual truth.

The fourth experiment checks the cost on answerable items. We report over-abstention and MMMU accuracy for every method, so that the helpfulness cost MoHoBench observed for DPO is measured, not assumed.