Asking When Vision and Dialogue History Disagree

Vision-language models (systems that answer questions about images in a running conversation) have a well-documented failure in multi-turn chat. Once a false statement enters the conversation, whether the user typed it or the model said it earlier, the model tends to stay consistent with that statement even when the image shows otherwise. This is called hallucination snowballing. Current fixes work silently inside the model: they push it to trust the pixels over the chat history. That is the right call when the history is wrong and the image is clear. It is the wrong call when the image is cropped, occluded, or blurry and the user actually knows something the camera cannot show. In that case, forcing the model to "look harder" amplifies noise.

We propose a model that, when it detects a disagreement between what it sees and what the conversation says, chooses among three actions. It can answer from the image, defer to the history, or ask the user a short targeted question such as "You mentioned a red mug, but I can't make out its color here. Can you confirm?" The choice depends on two cheap counterfactual measurements. The first is how much the chat history moves the answer. The second is how much the image itself moves the answer compared with a blank image. The choice also depends on who introduced the disputed claim, the user or the model. Each measurement costs one extra forward pass. The idea in two sentences: existing work either asks questions when the user's request is vague, or silently corrects vision-history conflicts. We instead treat a detected vision-history conflict as a reason to ask, and only when the image cannot settle it.

Core Claim

Snowballing and modality bias, meaning the model's habit of leaning on text over sensory input, have so far been treated as purely internal problems. Three lines of work fix them passively at decoding time:

All three assume the image contains the truth. The file's own diagnostics say this assumption often fails. In Diagnosing Vision-and-Language Navigation, agents succeed even when the objects named in the instruction are masked from view. In MLLMs are Deeply Affected by Modality Bias, Qwen2.5-VL gives the same MMMU-Pro answer with and without the image 56.53% of the time.

On the clarification side, the existing systems ask about something else:

  • CLAIM asks only about initial user underspecification.
  • Just Ask and DialFRED ask about the environment or task, not about a conflict between perception and history.
  • Clarification as Supervision uses clarification only as a training signal and removes it at test time.

Our claim is that conflict-triggered clarification, gated by an estimate of whether the image is sufficient, beats both "always trust the image" and "always conform to history". Specifically, it should give higher accuracy at a low question rate on conversations where the history is sometimes wrong and the image is sometimes insufficient. The gain should come specifically from the visual-sufficiency signal, not from generic uncertainty.

Research Program

At each turn the model has an image $v$, a dialogue history $h$, and a question $q$. We compute three answer distributions:

  • the normal one, $p(y \mid v, h, q)$;
  • a history-free one, $p(y \mid v, q)$, which RVD already computes;
  • a blank-image one, $p(y \mid \tilde v, q)$, where $\tilde v$ is a black image, as in the missing-modality stress tests of MLLMs are Deeply Affected by Modality Bias.

From these we define two quantities:

$$D_h = \mathrm{JSD}\big(p(y \mid v,h,q),\, p(y \mid v,q)\big), \qquad D_v = \mathrm{JSD}\big(p(y \mid v,q),\, p(y \mid \tilde v,q)\big)$$

Here JSD is the Jensen-Shannon divergence. $D_h$ measures how much the history is steering the answer. $D_v$ measures how much real visual evidence the image contributes beyond language priors.

The rule-based version of the policy works as follows:

  • If $D_h$ is small, there is no conflict, and the model answers normally.
  • If $D_h$ and $D_v$ are both large, the image actively contradicts the history. If the disputed claim came from the model's own earlier turn, it answers from the image and retracts the earlier claim. If the user introduced the claim, it answers from the image while stating the disagreement.
  • If $D_h$ is large and $D_v$ is small, the image cannot settle the question. The model asks a templated confirmation question about the disputed attribute if the user introduced the claim. If the model itself introduced it, the model abstains and retracts, because deferring to its own unsupported claim is exactly snowballing.

Provenance tagging, meaning tracking which speaker produced each claim, is trivial in our benchmarks because each turn has a known speaker.

The learned version replaces the thresholds with a small classifier. Its inputs are $D_h$, $D_v$, provenance, and optionally a linear probe on middle-layer activations, which SHARP found separate faithful from hallucinated states. The classifier is trained on a development split with a simulated user and a per-question cost, in the spirit of the clarification tax in Clarification as Supervision.

The simulated user answers from ground-truth annotations. It is a templated oracle, and that carries the same realism limitation DialFRED acknowledges. A small human-in-the-loop study checks whether the conclusions transfer to real users.

The main construction effort is evaluation data. MMHalSnowball and MM-Snowball contain hallucinatory histories over clear images, so "always trust the image" is optimal by design. From the same images we build a mixed split with three kinds of conversations:

  • the original false-history cases;
  • true-but-unverifiable cases, where the image is cropped or occluded so the queried attribute is no longer visible and only the history contains it;
  • false-history-plus-degraded-image cases, where neither source can be trusted and asking is the only route to a correct answer.

We use object annotations to choose crops that verifiably remove the target. This split is the main test, and its construction rules are fixed before any method is tuned.

Baselines

Naive references. The do-nothing reference is the base model, LLaVA-1.5-7B and Qwen2.5-VL-7B, decoding with full history. We also run two fixed policies: always ask on every turn, and never ask but abstain whenever $D_h$ is large. Always-ask sets the accuracy ceiling for a given question cost. Never-ask isolates how much value comes from asking versus merely refusing.

RVD. This is the strongest existing method, from Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models. It blends full-context and history-free predictions weighted by their divergence. It is the strongest competitor because it already uses our $D_h$ signal and is designed for this exact failure. The file reports two results for it:

  • LLaVA-1.5 accuracy on the HalluConv setting rising from about 15% to 33%, with an overall gain of over 24% in hallucinatory settings.
  • Accuracy above 90% on non-conflicting context-dependent queries.

We run RVD on the same backbones and splits, tuning its blending hyperparameter on the same development split with the same search budget as our thresholds. To beat it, we must match it within noise on original MMHalSnowball. We must then exceed it on the true-but-unverifiable and doubly-unreliable subsets at an ask rate of at most one question per three conflict turns. That budget is set loosely from the 0.71 questions per task that sufficed in DialFRED.

CAVR. We also run CAVR from MM-Snowball if code is available. The file reports no aggregate number for CAVR, only that it flattens the V-shaped decay, so we treat it as an additional silent-rectification competitor.

Uncertainty-triggered asking. This baseline is the Model Confusion heuristic from Just Ask. It asks when the gap between the top two answer probabilities under full context falls below a threshold, which is tuned on the same development split to the same ask rate as ours.

Prompted self-check. The same backbone is instructed to ask the user if the image and conversation disagree.

Ablations. Two ablations isolate the key ingredient. The first drops $D_v$, so the model asks whenever $D_h$ is large. The second drops provenance, so the model treats user and self claims identically. If dropping $D_v$ costs nothing, the visual-sufficiency idea is wrong.

Why not just…?

Why not just always trust the image, as RVD and CAVR do? On existing snowballing benchmarks this is close to optimal, which is exactly why those benchmarks cannot answer our question. This is a real threat. If degraded images still carry enough residual evidence that RVD answers correctly, our approach adds questions without adding accuracy. The true-but-unverifiable subset settles it. RVD's history-free branch there sees an image with the attribute removed and should fall back on language priors. Our prediction is that RVD scores below the history-deferring choice there, while our policy recovers it.

Why not just ask when the model is uncertain? Snowballing models are confidently wrong. The flip rate on imagination-type hallucinations in MMHalSnowball is nearly 100%, and LLaVA-1.5-7B falls below 2% accuracy in middle turns of MM-Snowball. Confidence computed under full history should therefore rarely flag the very cases that need a question. The matched-ask-rate comparison with the Model Confusion trigger tests this directly. If it matches us, the counterfactual passes are unnecessary.

Why not just prompt the model to notice the conflict and ask? It is the cheapest option, but the evidence is against it. Models conform to false history rather than question it, and models acting as judges of visual fidelity hallucinate errors of their own (Chen et al.). We include the prompted baseline and report its ask precision. A frontier-scale model doing well here would narrow the claim to open models of 7B scale.

Why not just let the agent look closer instead of asking? Active view selection (Learning Visually-Grounded Active View Selection for Embodied Question Answering) helps when better pixels exist. In our crop and occlusion cases the information is gone, so only the user can supply it. We regard "look versus ask" as the natural follow-up, with the $D_v$ signal deciding when looking is futile. We keep it out of scope here.

Decisive Experiments

Mixed-split comparison. The primary result is accuracy versus ask rate on the mixed split for both backbones, with all methods placed on the same curve by sweeping their thresholds. Our policy should dominate RVD and the confidence trigger on the two hard subsets. It should lose no more than about one point to RVD on the original false-history subset.

Diagnostic checks. The $D_v$ ablation must show a clear drop in ask precision, meaning the share of questions asked where an answer from the image would have been wrong. Without it, the core mechanism fails. We also check that turn-level accuracy along MM-Snowball's six-turn trajectories stays flat rather than V-shaped, and we report the added latency from the extra passes. As a sanity check that gating does not harm single-turn grounding, we run POPE, where no history exists and the policy should reduce to the base model.

Human-user check. A small study with real users answering the model's questions tests whether gains under the templated oracle survive natural replies.