Does Looking Again Replace Thinking in English? Visual Re-grounding for Native-Language Clinical Reasoning

When a vision-language model is asked a medical exam question in Hebrew or Japanese and told to "think step by step," it faces a choice that text-only research has already measured. In text, writing the intermediate reasoning in English usually beats reasoning in the question's own language. We propose that in image-based clinical questions, a large share of this English advantage is a perception problem rather than a logic problem. English reasoning may work better because the model's vocabulary for describing what it sees in an image was learned mostly from English captions. If so, a native-language reasoning chain that repeatedly re-inserts the image regions the model is attending to should recover most of the English advantage. It should do this without translation, and without the drift that happens when English reasoning loses the local clinical framing of the question.

The test is cheap and decisive. It needs an existing training-free method that splices attended image patches into a model's reasoning, a native, clinician-verified multilingual medical image benchmark, and a comparison across models that were or were not pre-trained on documents mixing many images and text. The practical stakes are concrete. Open models score near chance on non-English medical image exams, and the usual fixes, machine translation or English-only reasoning, are exactly the ones known to distort local clinical nuance.

Core Claim

The prediction is an interaction effect, not a main effect. Define the bridge gap for a language $\ell$ as

$$\Delta_\ell(g) = \mathrm{Acc}(\text{English rationale}, \ell, g) - \mathrm{Acc}(\text{native rationale}, \ell, g),$$

where $g$ indicates whether the reasoning chain is visually re-grounded. We claim three things. First, $\Delta_\ell(\text{on}) < \Delta_\ell(\text{off})$. Second, the shrinkage concentrates on questions that actually need the image and on scripts far from English, Hebrew most of all. Third, the shrinkage is largest in models pre-trained on interleaved image-text documents, meaning web pages or articles where many images and text passages appear in their natural order. Such models have learned to use an image that shows up in the middle of a sequence, so a patch spliced into a rationale is something they can read. If $\Delta$ does not shrink, the English bridge in multimodal clinical reasoning is about reasoning or medical knowledge, not perception. That negative result is also worth knowing, because it tells practitioners that re-grounding will not substitute for multilingual medical pre-training.

Where This Sits

Four lines of work bear on the claim, each in a different role.

The method comes from Interleaved-Modal Chain-of-Thought (ICoT). It introduces Attention-driven Selection, which reads the model's attention from a trigger token, such as a newline, to the image tokens and inserts a fixed budget of the most-attended patches back into the reasoning, with no retraining. It reports up to 14% relative improvement on the English M³CoT benchmark. ICoT never varies the language of the question or the rationale. We reuse its mechanism unchanged and ask a question it cannot answer: does re-grounding interact with reasoning language?

The motivating tension comes from two text-only papers and one multimodal one. Language models are multilingual chain-of-thought reasoners established on its multilingual grade-school math benchmark that English rationales beat native ones. Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought found that pure English reasoning causes topic drift on local entities in Korean, and proposed mixing languages within the chain. Both are text-only, so neither can separate a perception deficit from a reasoning deficit. WorldMedQA-V supplies the hint that images act as anchors: providing the image improved Spanish–English answer agreement. Its evaluation was direct prompting only, with no reasoning chains and no intervention for the Hebrew collapse it documents.

The regime and the constraint come from medical work. Towards building multilingual language model for medicine shows that training on rationales helps across six languages, but it is text-only. TumorChain shows that iteratively feeding cropped image regions back into clinical reasoning works in English 3D CT. It does so with an external segmentation expert and an English-only corpus. Detecting and Evaluating Medical Hallucinations in Large Vision Language Models shows that medical models readily invent image attributes. That constrains us to measure whether rationales describe real findings, not only whether final answers are right.

The moderator, interleaved pre-training, is motivated by evidence that models trained on isolated image-caption pairs do not improve with more in-context examples, while interleaved-trained ones do. This comes from Zebaze et al. on mOSCAR and from Small Visual Language Models can also be Open-Ended Few-Shot Learners. We extend that logic: a model that cannot use mid-sequence images should not benefit from mid-rationale patches.

Research Program

Stage 1: the core comparison. We cross reasoning language with re-grounding on WorldMedQA-V. It has 568 clinician-verified questions with images, drawn from licensing exams in Brazil, Israel, Japan and Spain, with English versions available.

Reasoning language takes four values:

  • direct answer;
  • native-language rationale;
  • English rationale with native question;
  • language-mixed rationale in the style of the Korean work, with English logic and native clinical terms.

Re-grounding is either off or on, using Attention-driven Selection with ICoT's default 64-patch budget and newline trigger.

We first run Qwen2-VL-7B and Chameleon-7B, both used by ICoT and both capable of interleaved input. We contrast them with LLaVA-next, which WorldMedQA-V already evaluated. We treat LLaVA-next as the "weakly interleaved" contrast because its training is dominated by single-image instruction data. That label is an assumption, which the controlled intervention below removes.

Stage 2: a controlled pre-training intervention. Model families differ in too many ways to isolate interleaved pre-training. To isolate it, we take one base model and run two matched continued-pre-training passes of equal token count:

  • one on interleaved documents from mOSCAR in the four target languages, with images kept in native document order;
  • one on the same images and text broken into isolated image-caption pairs.

Biomedical interleaved English documents from PIN-PMC are mixed into both passes equally. The prediction is that only the interleaved variant shows a large drop in $\Delta$ under re-grounding. If mOSCAR's Hebrew coverage proves too thin, we drop to the three remaining languages and report it.

Stage 3: error anatomy. For a stratified sample of rationales, native clinicians label each step on two axes:

  • error type: a misdescribed or invented image finding, a wrong medical fact, or a misreading of the question's local framing;
  • whether the step cites a finding actually present in the image.

The perception hypothesis predicts that re-grounding removes mainly the first error type, and mainly in native-language chains.

Baselines

Direct answering is the do-nothing reference. Each model answers in a single step with no rationale. It anchors how much any reasoning chain helps at all, and WorldMedQA-V already reports this regime: GPT-4o reaches 88% accuracy in Japanese but 58% in Hebrew, and LLaVA-next falls to 24–46%. For Qwen2-VL-7B and Chameleon-7B, no reported result was found on Lacuna for WorldMedQA-V, so we establish them ourselves.

English-rationale chain of thought is the strongest existing method for the question of reasoning language. The question stays in the native language and the reasoning is generated in English, following Language models are multilingual chain-of-thought reasoners, where PaLM-540B reached a 55% average solve rate on the grade-school math benchmark. That number is text-only math and is not directly comparable. We cite it only to identify the method.

English-rationale chain of thought with re-grounding is the strongest competitor overall. It combines that bridge with ICoT patch insertion. On English M³CoT, ICoT reports 44.1% zero-shot accuracy for Qwen2-VL-7B. Native re-grounded reasoning "beats" this competitor if it matches or exceeds it on the image-dependent subset for at least Hebrew and one other language, while scoring higher on clinician-judged fidelity to local framing.

All conditions share decoding settings, exemplar sets, the same 64-patch budget and the same trigger token. Prompt wording receives an equal tuning budget on a held-out split of 20% of the questions, and results are reported on the rest.

Two ablations isolate the key ingredient. The first inserts the same number of randomly chosen patches. This separates attention-chosen evidence from simply adding visual tokens. The second re-inserts the whole image at each step, which separates targeted re-looking from repeated global exposure.

Why not just…?

Why not just translate the question into English, reason with ICoT in English, and translate back? This is the most serious threat, and we run it as a condition. WorldMedQA-V reports that machine translation distorts local guidelines and terminology, and the Korean mixed-chain work found translation round-trips cause topic drift. Those findings come from settings without re-grounded reasoning, however. If translate-then-ICoT matches native re-grounding on accuracy, we will say plainly that translation is the practical winner. The perception claim would still stand or fall on the $\Delta$ interaction. To make the comparison fair, the clinician audit specifically checks whether translated pipelines answer a generic version of the question.

Why not just caption the image and reason in text? We include a condition where the model first writes a native-language description of the image and then reasons over its own caption with no further image access. ICoT showed that text-only rationales lose the link to visual evidence. TumorChain's reported 84.41% versus 44.04% for Qwen2.5-VL on its own benchmark is suggestive but confounded by training differences. Our condition settles it directly. If the caption pipeline closes $\Delta$ as well as re-grounding does, the effect is about having any visual description in the chain rather than re-looking.

Why not just fine-tune on rationales in each language? Towards building multilingual language model for medicine shows rationale training gains of up to 4.07% across languages. Native rationale data with images does not exist for Hebrew or Portuguese, though, and translated rationales carry the very distortions at issue. Fine-tuning is complementary rather than competing. Stage 2 tests the pre-training lever, and we flag rationale fine-tuning as follow-up work if re-grounding alone leaves a gap.

Proposed Experiments and Decisive Checks

Primary analysis. The primary analysis is a mixed-effects logistic regression of correctness on three factors and all their interactions, with random effects for question:

  • rationale language;
  • re-grounding;
  • interleaved-pre-training variant.

It is restricted to the image-dependent subset, defined as questions where removing the image lowers the model's direct-answer accuracy. WorldMedQA-V's Gemini Pro 1.5 results, with 4% to 27% gains from images, show this subset is substantial but uneven.

Power. With roughly 140 questions per language, a single language cannot detect small effects. We therefore run five sampled rationales per question and use paired tests, and we pre-register Hebrew versus pooled Roman-script languages as the key contrast.

What confirms the claim. The claim is supported if three things hold together:

  • the language-by-grounding interaction is negative and significant;
  • random-patch insertion does not reproduce the effect;
  • the interleaved continued-pre-training variant shows a larger interaction than its pair-only twin.

The error anatomy should show re-grounding cutting invented-finding steps in native chains more than in English ones.

What refutes it. The claim fails if $\Delta$ stays flat while re-grounding helps both languages equally. That would mean the English bridge in clinical image reasoning lives in knowledge, not perception, and the right investment is multilingual medical pre-training, not inference-time grounding.

Secondary measures. Two secondary measures are cross-language answer agreement between each native question and its English version, extending WorldMedQA-V's observation, and token cost per question. The cost measure matters because re-grounding adds 64 patches per step.