Ask a vision-language model (a model that reads an image and a text prompt and answers in text) about a chart or a clinical tracing over several turns, and its answers get worse as the conversation gets longer. The model starts trusting what was said about the figure more than the figure itself. If an earlier turn claimed "the blue series peaks in 2018," the model tends to repeat that claim even when the peak is in 2016. Removing the conversation does not fix this, because later questions depend on it. "What happens to that peak if we drop the outlier?" makes sense only given earlier turns. We propose a simple split. Dialogue history may decide which part of the figure a question refers to and what conclusions to draw. It may never decide what the figure shows. That last job goes to a fresh look at the image, made without the conversation in view.
Concretely, at each turn the model first rewrites the user's question so it stands alone. This step resolves pronouns, omitted words, and phrases like "the orange line" into explicit visual queries. The same frozen model then answers those queries from the image alone, with no dialogue history. The answers go into a small ledger of verified visual facts, and that ledger outranks anything earlier turns asserted about the image. We test this on multi-turn dialogues over technical figures: charts, scientific plots, competition-math diagrams, and a medical imaging slice. These dialogues contain two kinds of planted false claims, one about what the figure shows and one about what follows from it. The new part is the prediction that the two kinds fail differently and need different fixes. Existing remedies that boost the image's influence during decoding should help with the first kind and much less with the second.
Core Claim
Multi-turn hallucination over technical figures has two separable sources. The first is perceptual drift: the model repeats a wrong description of the image that came from the history. The second is inferential drift: the model accepts a wrong domain-level conclusion about a correctly read image. Perceptual drift can be largely removed, without training, by reading the image again with the history hidden, at exactly the points where the dialogue refers back to the figure. Inferential drift cannot be removed this way, and it cannot be removed by methods that contrast a "vision-only" prediction with a full-context one. Those methods assume the image alone contains the answer, and in technical domains it does not, because the answer also requires applying a theorem or a clinical rule. The second, testable claim is that hiding history from perception only does not erase the benefit dialogue provides when the history carries legitimate, disambiguating information.
What Prior Work Establishes and Leaves Open
Three papers motivate the perceptual-drift half of the problem, and all three study natural photographs.
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models shows that one planted false turn makes LLaVA-1.5 and mPLUG-Owl2 lose at least 31% accuracy. It fixes this with Residual Visual Decoding (RVD), which blends full-context and vision-only output distributions.
- MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue extends this to six-turn dialogues. It finds a V-shaped accuracy curve in which LLaVA-1.5-7B falls below 2% in middle turns and then recovers when explicitly told to re-examine the image. The authors read this as "visual fading," not forgetting.
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow measures up to a 60% drop in attention to image tokens by turn 20, and relays image tokens between agents to counter it.
All three test objects, colors, and spatial relations in photographs. RVD and CAVR also rely on a vision-only branch that can answer by itself. We keep their diagnosis but change the setting to figures where answers need domain reasoning. We also replace logit blending with an explicit perception call that excludes the history. The V-shaped recovery in MM-Snowball is direct evidence that the model still has the visual information and simply needs to be made to consult it.
The mechanism comes from the query-rewriting literature. McQueen: a Benchmark for Multimodal Conversational Query Rewrite shows that references in visual dialogue often point at image entities, and rewrites them into standalone queries with a pointer-copying model. It stops at the rewrite, and its images are everyday VisDial scenes. We use the rewrite as the trigger for re-perception and target technical figures. Where the rewrite says the user means "the orange line at x = 2015," the next step reads that value from pixels, not from the transcript. You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding supplies a design lesson. On text-only developer dialogues, 70–92% of agent failures happen because the right target is never surfaced, while selection is about 90% accurate once it is. We therefore report reference-resolution errors separately from perception errors instead of folding them into one accuracy number.
Two lines of work constrain what we can claim.
- Perfect perception does not fix domain reasoning. GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level gives models human-written transcriptions of the diagrams and still sees near-chance accuracy (best 26.22% against a 25% random baseline). Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset attributes 42.2% of GPT-4V's errors to reasoning and 31.9% to recognition. So we do not claim to close the domain-reasoning gap. We claim to stop the conversation from adding perceptual errors on top of it.
- Supplying definitions does not teach application. TheoremQA: A Theorem-driven Question Answering Dataset finds that giving models the relevant theorem does not significantly help. This is why the proposal adds no retrieval of domain knowledge.
Two papers frame the evaluation.
- Beyond Accuracy: Evaluating Visual Grounding in Multimodal Medical Reasoning tests single-turn medical question answering with blank and shuffled images. It finds that reinforcement learning on final-answer rewards produces fluent visual claims unrelated to the image: 38–43% of visual claims are ungrounded, and Image Sensitivity falls from 48.2% to 39.8%. We apply those counterfactual metrics turn by turn in dialogue.
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details shows that chart accuracy falls from up to 79% to the 30% range once title shortcuts are removed. It also reports collective accuracy, meaning all elementary questions about one chart answered correctly, below 25%. So history-blind perception will itself be imperfect, and we measure its error rate directly.
Finally, there is an apparent contradiction to address. Advancing Conversational Diagnostic AI with Multimodal Reasoning reports that analyzing images within dialogue improves diagnosis, the opposite of the snowballing papers. Our reading is that AMIE's dialogue mostly supplies where to look and what context applies, while the snowballing benchmarks inject claims about what is visible. The proposal makes this distinction operational, and one experiment tests whether it resolves the tension. That paper also reports that supervised fine-tuning hurt flexibility compared with a frozen model plus explicit state tracking. This supports keeping our method training-free.
Research Program
The method. We call it History-Blind Re-Perception. It runs on any frozen vision-language model in three steps per turn.
- Resolve. With the full history and image in view, the model rewrites the current user turn into a standalone question. It also lists the elementary visual facts the answer depends on, phrased so each can be checked without the conversation. An example is "y-value of the orange series at x = 2015," with an optional bounding region. Prior assistant turns are scanned the same way, and any perceptual assertion they made is extracted as a candidate fact.
- Perceive. Each fact query is sent to the same model with only the image, or a crop of the named region, and the query text. No history is included. Answers go into a ledger. Where a ledger entry conflicts with an earlier assertion, the conflict is marked explicitly.
- Reason. The model answers with full history plus the ledger, under the instruction that ledger entries override any prior statement about the image. Domain assumptions, hypotheses, and user-supplied constraints from the history remain untouched.
The cost is one rewrite call, a few short perception calls, and one answer call per turn. We control for that cost in every comparison.
The benchmark. We will build a dialogue benchmark over technical figures, because none in the file combines references, technical artifacts, and planted false claims. The working name is TechRef-Dialog.
- Figure sources. Charts come from ChartQA (ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning), whose underlying data tables make perceptual answers checkable by program, and from Chart101-style synthetic plots, where ground truth is exact. Scientific figures come from VeriSciQA. Diagrams for turns that require theorem application come from MATH-Vision. A medical slice draws on SLAKE and VQA-RAD.
- Dialogue structure. Each dialogue has four to eight turns. References are made through visual attributes such as color, position, ordinal rank, and "the spike we discussed."
- Three variants per base dialogue. A clean variant has correct history, including at least one turn that supplies needed context, so it measures whether dialogue still helps. A perceptual-injection variant plants a false claim about what the figure shows. An inferential-injection variant plants a plausible but false domain conclusion about a correctly described figure, such as a wrong monotonicity argument or a misapplied clinical threshold.
- Construction and checking. Dialogues are generated from templates over the ground-truth tables and then paraphrased. Humans verify a stratified subset for naturalness and answer correctness. Every turn is labeled for whether it requires history and whether it requires domain inference.
The analysis. Beyond accuracy, we measure the following per turn.
- Flip rate: how often a planted claim changes an answer that was correct without it.
- Reference-resolution accuracy: whether the rewrite picked the right figure element.
- Ledger accuracy: whether the history-blind perception call read the element correctly.
- Image Sensitivity and Blank Drop: the counterfactual metrics from Beyond Accuracy, computed per turn.
These separate the three ways the pipeline can fail: wrong reference, wrong reading, or wrong reasoning on correct facts. We also track attention to image tokens across turns, using the layer-wise ratio introduced in LLaVA Steering. This tests whether the ledger restores visual attention or bypasses the need for it.
Baselines
Full-context answering. The reference point is the unmodified model answering each turn with full history and image. It shows how much snowballing occurs on technical figures and is the denominator for every gain.
Re-examine prompting. The model gets full history plus an instruction to look at the image again before answering. This mirrors the final-turn recovery in MM-Snowball and costs almost nothing. If it closes most of the gap, the ledger is unnecessary. Our method must beat it on perceptual-injection dialogues by a margin larger than seed-to-seed variation.
Residual Visual Decoding (RVD). This is the strongest existing method for this question. It is training-free, targets exactly history-induced visual error, and runs on the same open models. On MMHalSnowball it raised LLaVA-1.5 accuracy in hallucinatory conversations from about 15% to 33%, and it improved accuracy by over 24% across models (Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models). CAVR from MM-Snowball is the other natural competitor. The file reports that CAVR achieves the best Snowballing Resistance Score but gives no number, so no reported result was found on Lacuna for CAVR's score, and we will run it ourselves where code permits. No reported result was found on Lacuna for any method on multi-turn technical-figure dialogue, because the benchmark does not exist yet.
Fairness and the bar to clear. All decoding baselines run on the same models (LLaVA-1.5-7B to match the RVD paper, plus InternVL-Chat and GPT-4o through its API where logits are not needed). They use the same dialogues and the same hyperparameter search budget on a held-out development split. To match our method's extra forward passes, they also get a matched-call self-consistency variant. Beating them means three things: lower flip rate on perceptual-injection dialogues, no loss on clean dialogues, and a gap that survives the matched-call control.
Rewrite-then-forget. The model rewrites the question to standalone and answers it from the image with no history at all. This is the strongest simple alternative and the key ablation (see below).
Isolating the ingredients. Two further ablations each remove one piece of the method. Rewrite-without-blindness keeps the McQueen-style rewrite but answers with full history. It tests whether hiding history from perception matters. Blindness-without-rewrite re-asks the raw user turn without history. It tests whether resolving references matters.
Perception-to-text pipeline. We extract the chart table with OCR and give the table plus dialogue to a text-only model. This represents the "convert to text" school.
Why not just…?
Why not just tell the model to re-examine the image every turn? MM-Snowball's own V-shaped curve shows this works in the turn where the prompt appears. That makes it a serious competitor on perceptual drift. Our bet is that the prompt still leaves the false claim in the context the model attends to while looking. Hiding it removes the competition entirely, and visual attention erodes over longer dialogues (Visual Multi-Agent System). The re-examine baseline is run at matched cost. If it equals our method on perceptual injections, we will report that the ledger adds only auditability and narrow the contribution to the reference-resolution step.
Why not just rewrite each question to standalone and drop the history? This is the most likely way the idea could collapse. On perceptual turns it may perform as well as our method. The proposal survives only if the clean-dialogue condition shows a cost. Turns that depend on user-supplied constraints or earlier agreed conclusions should fail when history is discarded, as AMIE's finding that dialogue helps diagnosis suggests. If rewrite-then-forget matches us on clean dialogues too, the honest conclusion is that history is not needed for these tasks, and we would report the simpler method as the answer.
Why not just contrast against a vision-only prediction, as RVD and CAVR do? These methods are strong on photographs, where the image alone answers "is there a dog." On a turn like "given that slope, is the function convex on this interval," a vision-only pass has no access to the needed constraint or reasoning. Contrasting against it should then push the answer toward a naive visual guess rather than toward the truth. This is a prediction, not a result. The inferential-injection split tests it directly: we expect RVD's gain there to be small or negative, while it stays competitive on perceptual injections.
Why not just extract the chart to a table and let a text model converse? ChartQA already found that table-based models fail when questions name visual attributes like "the orange line." Those are exactly the references conversations use. Scatter plots, where Plot Twist reports accuracy as low as 4.2%, do not reduce cleanly to tables either. We include the pipeline anyway, since on bar charts with clean labels it may be competitive. A per-figure-type breakdown will show where it holds up.
Proposed Experiments and Decisive Checks
The main experiment runs every baseline and our method on all three dialogue variants, all figure sources, and all models. It reports per-turn accuracy, flip rate, and per-turn Image Sensitivity, with confidence intervals from bootstrap resampling over dialogues.
First decisive check: the dissociation. On perceptual-injection dialogues, history-blind re-perception should cut flip rate well below full-context answering and at least match RVD. On inferential-injection dialogues, neither our method nor RVD should recover much, because both methods only correct perception, and GeoSQA and TheoremQA indicate the remaining failure is reasoning. If our method does substantially fix inferential injections, the two-source account is wrong. Planted domain claims would then be acting through perception after all, and we would revise the claim.
Second check: the dialogue tension. On clean dialogues with helpful history, our method should stay at or above full-context accuracy, while rewrite-then-forget falls on history-dependent turns. This is the test of whether separating "where to look" from "what is there" lets dialogue help, as in AMIE, without letting it hurt, as in the snowballing papers.
Third check: grounding. With the image blanked or swapped for a different figure, ledger entries should change and downstream answers should follow. That is, per-turn Image Sensitivity under our method should exceed full-context answering's. If answers stay fixed when the image is blanked, the ledger is producing confident visual claims unrelated to the image, the failure mode Beyond Accuracy documents.
Fourth check: locating the remaining error. The error breakdown into reference, perception, and reasoning should show which bottleneck remains. Given Chart101's sub-25% collective accuracy, we expect perception-call errors to bound the method's ceiling. Reporting that bound is part of the result.
References
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
- MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
- McQueen: a Benchmark for Multimodal Conversational Query Rewrite
- You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- TheoremQA: A Theorem-driven Question Answering Dataset
- Beyond Accuracy: Evaluating Visual Grounding in Multimodal Medical Reasoning
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Advancing Conversational Diagnostic AI with Multimodal Reasoning
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering