When a vision-language model, a language model that also reads images, answers a medical imaging question step by step and points to a region ("the opacity in the lower left lobe"), we tend to treat that pointer as an explanation. Nothing in current methods checks whether the pointer matters. If the cited region were erased, would the answer change? If an irrelevant hospital watermark were changed, would the answer stay the same? This proposal makes those two questions the training signal. At test time, with no labels, we adapt a small set of prompt vectors so that three things hold. The model's answer should depend on the region its own reasoning cites. It should not depend on an equally sized random region. It should not depend on nuisance changes such as scanner contrast, compression, or watermarks. The claim is that this pairing of sensitivity to cited evidence with invariance to nuisance closes part of the gap generalist models show on specialized images under distribution shift. Invariance alone, which existing test-time methods enforce, cannot close that gap, because a model that ignores the image is already perfectly invariant.
The opportunity comes from a specific diagnosis. On the Out-Of-Distribution Generalization of Large Multimodal Models shows that on medical and molecular datasets, zero-shot multimodal models sit near chance (about 50% on binary tasks). A linear classifier trained on the same frozen image features reaches 0.830 accuracy on COVID-CT. The paper calls this a "mapping deficiency": the perceptual information is present, but the model does not connect it to specialized labels. A model that answers from language priors rather than from the image will produce exactly this pattern. That makes the question of whether the cited evidence causally drives the answer the right lever to pull.
Core Claim
Grounded chain-of-thought, meaning step-by-step reasoning in which each step names an image region with a bounding box, is mostly decorative in specialized domains under shift. Making the cited region causally necessary relative to a matched control region is a label-free objective that improves both accuracy and faithfulness under shift. The proposal has two parts. The first is a measurement: a visual counterfactual audit of multimodal medical reasoning. The second is a method: counterfactual test-time adaptation. The audit is valuable even if the method fails.
How This Relates to Prior Work
Three papers motivate the proposal by showing that reasoning text detaches from evidence. Right Diagnoses, Decorative Reasoning edits clinical text questions and finds a 72.9% chain-decoupling rate: reasoning fails to register input changes or fails to mediate the answer. Corrupting chains changes accuracy by a median of about 0 points. That audit is text-only. We carry its question-edit logic into the image, where the edit is "remove the evidence" rather than "flip a negation." Don't Blink shows that attention to evidence boxes decays during reasoning in 83.5% to 100% of samples. Its remedy is a veto that rejects confident answers. We instead try to change the model so the evidence keeps mattering. HallusionBench uses human-edited control image pairs on natural images and finds that models default to memorized knowledge; GPT-4V reaches only 31.42% question-pair accuracy. Our edits are automatic, medical, and driven by the model's own citations.
Two papers supply the method we build on. Grounded Chain-of-Thought trains models to emit boxes per reasoning step. It also finds that larger models score higher on answers but lower on answer-grounding consistency, which suggests boxes can be produced without being used. It needs box supervision on natural images. CoTBox-TTT is the closest existing work. It predicts a box, crops it, and adapts soft prompts so predictions on the global image and the crop agree, with LLaVA gaining 12.3% closed-ended accuracy on PathVQA. Our method differs in one essential way. CoTBox-TTT enforces only agreement, which a prior-driven model already satisfies. We add a term that rewards the answer for changing when the cited region is erased, and we calibrate that term against erasing a random region. We also apply the objective per reasoning step rather than to a single box.
TumorChain constrains our design rather than supplying it. It achieves strong out-of-distribution results on DeepTumorVQA (73.30% lesion recognition) by training on 1.5 million curated CT instructions. That route is unavailable for long-tail domains, which is why we restrict ourselves to label-free test-time adaptation.
Research Program
The first component is the audit. For each image-question pair, the model produces a grounded chain whose steps cite boxes. We then build three edited images. The cited ablation fills the cited region using a fixed fill (local mean, blur, or a patch copied from elsewhere in the same image). The control ablation applies the same fill to a random region of equal area that does not overlap the cited one. The nuisance edit changes intensity, compresses the image, alters resolution, or adds or removes a watermark, imitating device and site shift. Because the cited and control ablations use the same fill, artifacts from the fill itself cancel when we compare them.
We report a visual chain-decoupling rate, in the spirit of the text audit: the fraction of cases where cited ablation changes neither the answer nor the chain more than control ablation does. We also report a nuisance-sensitivity rate: the fraction of answers that flip under nuisance edits. On a 300-image subset we will have a clinician mark the decisive region. This lets us distinguish two failures: decorative citations, where the model cites the right region but does not use it, and misdirected citations, where it cites the wrong region.
The second component is the adaptation. Let $p(\cdot\mid x)$ be the model's answer distribution on image $x$ after generating its grounded chain. Let $n(x)$, $a_r(x)$ and $a_c(x)$ denote the nuisance-edited, control-ablated and cited-ablated images, and let $D$ be the Jensen–Shannon divergence. With the image encoder and language model frozen, we update only soft prompt vectors to minimize
$$ \mathcal{L}(x) = D\big(p(\cdot\mid x),\,p(\cdot\mid n(x))\big) + D\big(p(\cdot\mid x),\,p(\cdot\mid a_r(x))\big) - \lambda \min\Big(D\big(p(\cdot\mid x),\,p(\cdot\mid a_c(x))\big) - D\big(p(\cdot\mid x),\,p(\cdot\mid a_r(x))\big),\; m\Big). $$
Here $\lambda$ weights the sensitivity term and $m$ is a margin that caps the reward so the model cannot be pushed into flipping arbitrarily. Each term blocks a specific way of cheating. The random-control term blocks "change the answer whenever anything is erased." The nuisance term blocks reliance on site artifacts. The capped sensitivity term blocks the blind, prior-only solution. Cited regions are recomputed each adaptation step, so the model can move its citation toward evidence that actually matters. For multi-step chains we apply the sensitivity term to each step's box and average.
The third component treats the same quantity as a hallucination signal. The necessity score, the cited-ablation divergence minus the control-ablation divergence, needs no training. Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distribution shows that probes trained on internal activations collapse to an AUC of about 0.50 across datasets because they learn task format. A behavioral intervention score has no format to latch onto. Whether it transfers better is a direct test of the open question that paper leaves.
Baselines
The naive reference is the frozen model answering directly without reasoning, together with the same model using plain step-by-step prompting. These two together show whether reasoning helps at all here. The text audit found that chain-of-thought lowered Llama-3.1-8B accuracy by more than 12 points, so we expect plain reasoning may not beat direct answering.
The second baseline is grounded chain-of-thought prompting with no adaptation. It isolates whether the gains come from emitting boxes or from adapting on counterfactuals.
The third baseline is 8-shot in-context learning, which On the Out-Of-Distribution Generalization of Large Multimodal Models reports raising GPT-4 on XCOVFour from about 45% to over 84%. It uses labeled examples, which our method does not. It is still the most effective cheap remedy in the file. We run it on the same open models with the same examples, and we report it both on standard splits and on a split where watermarks correlate with labels in the examples but not at test.
The strongest competitor is CoTBox-TTT, reimplemented on LLaVA and Hulu-Med as in its paper. For LLaVA it reports a closed-ended accuracy gain of 12.3% on PathVQA and 8.09% on VQA-RAD, and an open-ended recall increase from 7.74 to 32.60 on PathVQA. Our method uses three edited views per step, so we give CoTBox-TTT the same number of forward passes and the same 20-epoch soft-prompt budget. We tune both on the same small held-out set. Beating it means higher closed-ended accuracy on PathVQA and VQA-RAD at matched compute, plus a clearly larger margin on nuisance-shifted versions of those sets. We would also expect a lower visual chain-decoupling rate.
The key ablations remove the sensitivity term, which leaves a pure invariance method close in spirit to CoTBox-TTT, and remove the random-control term. If removing sensitivity costs nothing, the core claim fails.
A labeled linear probe on frozen image features serves as an upper reference. It shows how much discriminative information the features contain. It is not a competitor, because it uses labels.
Why not just…?
Why not just use a larger model? Grounded Chain-of-Thought finds that 72B models have lower answer-grounding consistency than 7B models. The out-of-distribution study finds larger models sometimes worse on TerraInc and SVIRO. We will still run the audit on the largest open model we can serve. If scale alone drives the decoupling rate down, our method matters less, and we will report that.
Why not just give a few labeled examples? This is a real threat on clean splits, where its reported gains are large. The same study notes that in-context learning latches onto spurious cues such as hospital watermarks and fails on DrugOOD. Our watermark-correlated split tests exactly this. In-context learning also needs labels from each new site; our method does not.
Why not just inject domain knowledge into the prompt? TheoremQA finds that adding theorem definitions gives no significant boost, and the out-of-distribution study finds expert background text has very limited impact. The deficiency is in connecting what is seen to what is known, not in missing definitions. We include a knowledge-prompt baseline anyway, because it costs little.
Why not just plan before looking? Thinking Before Looking gains 31.74% on MMVP for Gemini 1.5 Pro. Its gain on MathVista is about 3.15%, however, and it fails on smaller open models. Medical findings require inspection before any plan is possible. We run it on HallusionBench and on the medical sets to check.
Why not just reject unreliable answers with a vision veto or entropy? Abstention is complementary rather than a fix. Don't Blink also reports that text entropy cannot see evidence collapse. We compare accuracy at matched coverage, and we compare the necessity score's cross-dataset detection AUROC against full-response entropy.
Proposed Experiments
Models are LLaVA, Hulu-Med and Qwen2.5-VL-7B. Medical evaluation uses VQA-RAD, SLAKE and PathVQA, each in original and nuisance-shifted form, along with COVID-CT and XCOVFour from the out-of-distribution suite to test the mapping-deficiency setting directly. HallusionBench checks that the method does not hurt natural-image control pairs.
There are four decisive checks. The first asks whether grounded medical chains show a visual decoupling rate comparable to the 72.9% text figure; this would establish the problem. The second asks whether the full objective beats CoTBox-TTT at matched compute, especially under nuisance shift. The third asks whether removing the sensitivity term erases that advantage; this isolates the new ingredient. The fourth asks whether the necessity score, computed on VQA-RAD, keeps its error-detection AUROC on PathVQA and SLAKE, where activation probes would be expected to fall toward 0.50.
We expect failure on diffuse pathologies that no box captures, a limitation shared with CoTBox-TTT. We will report results on that subset separately rather than averaging it away.
References
- On the Out-Of-Distribution Generalization of Large Multimodal Models
- Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought
- Don’t Blink: Evidence Collapse during Multimodal Reasoning
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- Grounded Chain-of-Thought for Multimodal Large Language Models
- CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training
- TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis
- Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distribution
- TheoremQA: A Theorem-driven Question Answering Dataset
- Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination