Image-Contingent Text Bridges for Medical Visual Reasoning

Medical vision-language models can score well on visual question answering while barely looking at the image. A counterfactual audit found that a model trained with reinforcement learning on text-only medical questions keeps 81% of its VQA-RAD accuracy when the X-ray is replaced by a blank gray square. The same model does slightly better on PathVQA when given a mismatched slide. One promising family of fixes splits the model in two. A "perceiver" looks at the image and writes down what it sees, and a separate text-only "reasoner" answers the question from that written description. This split helps on math and chart reasoning. In medicine, however, it has two weak points. The first is that the check for a good description ("can a blind reasoner solve the problem from it?") passes automatically when the question alone already gives away the answer. The second is that the description is written in clinical jargon and abbreviations, which general-purpose tokenizers break into meaningless fragments before the reasoner ever reads it.

We propose to train the perceiver with a reward that only pays for information the image actually contributed. The reward is the blind reasoner's accuracy given the perceiver's findings, minus its accuracy given the question alone or given findings written about a different image. We pair this with a controlled experiment on the vocabulary of the bridge text: free-form findings, findings normalized to expanded canonical clinical terms, and findings read by a reasoner whose tokenizer has been extended with domain terms. The proposal therefore makes two claims. One is a training method meant to raise accuracy without grounding collapse. The other is a direct test of whether domain-vocabulary mismatch damages the text interface in decoupled multimodal reasoning. If it does not, that negative result is informative in its own right.

Core Claim

The perception–reasoning split is the right architecture for medical visual reasoning only if the written findings are forced to depend on the image. Today's perception rewards do not enforce this dependence in domains with strong text priors. Our claim is that a contrastive, image-contingent reward on the findings will match the strongest reinforcement-learning baseline on medical VQA accuracy while sharply increasing how much the answers depend on the image. We also claim that whether the findings use fragmented or normalized vocabulary measurably affects how often the bridge fails. Fragmentation here means the tokenizer cutting a term like bronchovasculature into seven sub-pieces. The first claim concerns the reasoning direction. The second takes the vocabulary direction and moves it from the setting where it has been studied, text-only report summarization, to the interface where we think it matters most.

What the Literature Establishes and Leaves Open

Three papers motivate the reasoning side. MATH-Vision and GAOKAO-MM find that models usually recognize diagram elements correctly and then fail at the deduction that follows. GeoSQA replaces diagrams with human-written descriptions and finds that reasoning still fails. The bottleneck therefore sits after perception.

Three papers supply the decoupled method we build on. Caption This, Reason That shows that having a model caption the image before reasoning helps a lot. It also shows that feeding the image back alongside the caption often hurts, so our reasoner sees only text. RAPID trains a perceiver with reinforcement learning to write query-relevant captions for a frozen text reasoner and reaches 53.2% average on a multimodal math suite. Bad Seeing or Bad Thinking?, called MoCA below, rewards a <recognition> block whenever a blindfolded text LLM can solve the task from it. None of the three was evaluated on clinical images. None of them asks whether the description carries information beyond what the question already reveals.

The medical evidence motivates and constrains the design. MedVLThinker finds that reinforcement learning with verifiable rewards (answer-correctness rewards, optimized with Group Relative Policy Optimization, or GRPO) beats supervised fine-tuning. It also finds that RL on text-only data beats RL on image–text data. Beyond Accuracy explains why this happens: those gains come largely from text priors. Image sensitivity falls under RL, and 38–43% of reasoning traces contain hallucinated visual claims. Its blank- and shuffled-image protocol is exactly the counterfactual we turn into a training signal. Beyond Accuracy diagnoses the problem but proposes no training fix. Our method is that fix, and it reuses their metrics as the yardstick. How Do Medical MLLMs Fail? shows that medical models attend poorly to clinically relevant regions. Its inference-time intervention, VGRefine, masks out irrelevant image tokens. That approach is complementary to ours and can be stacked on top.

On vocabulary, Specialised or Generic? shows that general tokenizers spend about 2.7 tokens per radiology word against about 1.0 for a domain tokenizer. For models trained from scratch, report summarization BLEU-1 rises from 0.162 to 0.273 with the domain tokenizer, and the authors explicitly leave multimodal tasks open. Biomedical Language Models are Robust to Sub-optimal Tokenization cuts the other way. Better morpheme segmentation barely changes encoder NER (93.0 vs. 93.4 F1 on BC5CDR-chem) because higher layers repair the fragments. MeDAL documents how ambiguous clinical abbreviations confuse general models. We take the robustness result seriously as the null hypothesis. A large pretrained reasoner may well repair fragmented jargon too, and our vocabulary experiment is designed so that this outcome would show up clearly.

Research Program

The system has two parts. The perceiver is Qwen2.5-VL-7B, which receives the image and question and writes a short findings block. The reasoner is Qwen3-8B, the reasoner used in RAPID. It stays frozen, never sees the image, and answers from the question plus findings.

The perceiver is trained with GRPO on the image–text medical training data used by MedVLThinker. Its reward is image-contingent. Let $q$ be the question, $f(x)$ the findings the perceiver writes for image $x$, and $\tilde{x}$ a randomly drawn image from the same benchmark and modality. Let $A(\cdot)$ be the reasoner's accuracy averaged over $k$ sampled answers. The perception reward is

$$ r_{\text{perc}} = A(q, f(x)) - \max\big(A(q, \varnothing),\ A(q, f(\tilde{x}))\big), $$

where $\varnothing$ means the question with no findings. This reward is zero when the findings add nothing the question did not already reveal, and it is negative when findings about the wrong image work just as well.

The question-only term $A(q,\varnothing)$ is a per-question constant computed once. The shuffled-image term requires an extra perceiver rollout, which we compute on a fixed fraction of steps to bound cost. Because both terms use the same reasoner, a reasoner that cannot solve the problem at all yields a reward of zero rather than a penalty. This blunts the false-negative problem MoCA reports for its blindfolded proxy. The final reward adds answer correctness to $r_{\text{perc}}$, plus RAPID's penalty against empty captions.

The vocabulary study runs on the same pipeline under three conditions for the bridge text. In the free-form condition, the perceiver writes whatever it likes. In the normalized condition, the perceiver is rewarded for, and post-processed toward, expanded canonical terms from a fixed clinical lexicon, with abbreviations spelled out. In the extended-tokenizer condition, the free-form findings are read by a reasoner whose vocabulary has been extended with high-frequency clinical terms. Embeddings for the new terms are initialized from the mean of their sub-pieces and briefly trained on medical text, following the expansion approach of Gold Panning in Vocabulary.

Alongside these conditions we run a controlled perturbation on held-out examples where the reasoner already succeeds. We rewrite the findings into abbreviated or more fragmented forms that carry the same meaning, and we measure the drop in reasoner accuracy. We also stratify bridge failures by the tokens-per-word ratio of the key clinical terms they contain. Together these tests ask whether the encoder-style "repair" survives in a generative reasoner that reads machine-written jargon.

Baselines

The naive reference point is Qwen2.5-VL-7B answering directly, with and without its own self-caption, evaluated under the full Beyond Accuracy protocol. This gives the accuracy floor, and it gives the image-sensitivity baseline that Beyond Accuracy reports as 48.2% on average before any RL.

The strongest existing accuracy competitor is MedVLThinker-7B trained with RL on text-only data. Its reported average accuracy is 54.88% across PMC-VQA, MMMU-Health, MedXpert-MM, PathVQA, SLAKE, and VQA-RAD. It is the strongest competitor because it is the best published open 7B result on this exact suite. It is also the model that Beyond Accuracy shows to be shortcut-driven, so it is the natural target for a claim about accuracy with grounding. We run the public checkpoint and, as a control, retrain it on our compute budget with its data and difficulty filter. Our method beats it if it matches its six-benchmark average within noise while raising image sensitivity above the 48.2% pre-RL level and increasing the blank-image drop on VQA-RAD. That benchmark is where text-only RL keeps 81% of its accuracy on blank images and where both RL variants reach 63%.

The strongest decoupled competitors are a RAPID-style perceiver, trained with RAPID's own reward, and a MoCA-style perceiver, trained with the blindfolded-sufficiency reward. Both use the same Qwen3-8B reasoner, the same medical training data, the same number of GRPO steps, and the same GPU-hour budget. Our extra reasoner calls count against that budget, and we reduce rollout counts to compensate. No reported result was found on Lacuna for RAPID or MoCA on medical benchmarks. RAPID's 53.2% applies to the math suite only. These comparisons isolate the key ingredient, the counterfactual subtraction, because everything else is held fixed. A further ablation keeps our full reward but removes the shuffled-image term, keeping only the question-only term. Another ablation crosses each reward with each vocabulary condition so the two effects can be separated.

As a reference for an inference-time fix, VGRefine is reported to add 5.6 points on VQA-RAD and 11.3 on PathVQA for HuatuoGPT-V-7B, and to raise OmniMedVQA from 71.3% to 74.4%. We apply it to our perceiver to test whether the two gains add.

Why not just…?

Why not just run end-to-end RL with verifiable rewards, as MedVLThinker does? This approach gives the best accuracy, but Beyond Accuracy shows how it gets there. Image–text RL lowers image sensitivity to 39.8%. Text-only RL scores slightly better with mismatched images on PathVQA (a Visual Reliance Score of −0.09). A correctness-only reward cannot tell an answer the model saw apart from one it guessed. Our head-to-head comparison on both accuracy and the counterfactual metrics settles this. If MedVLThinker matches us on both, the proposal fails.

Why not just use RAPID or MoCA unchanged on medical data? This is the most serious threat. These methods already route reasoning through image-derived text, so they might already ground answers in the image. The reason to doubt it is that their reward asks only whether the text suffices. On VQA-RAD, where blank images keep 81% of accuracy, a description that merely restates the question would usually pass. The matched-budget comparison answers the question directly. If RAPID and MoCA score as high as our method on image sensitivity and blank drop, the counterfactual term is unnecessary and we will say so.

Why not just swap in a domain tokenizer? Retokenizing a pretrained VLM is expensive and risks damaging what it already knows. The encoder evidence in Biomedical Language Models are Robust to Sub-optimal Tokenization suggests the gain may be small. The BLEU gap in Specialised or Generic? comes from models trained from scratch, not adapted ones. For these reasons we confine vocabulary changes to the bridge: lexicon normalization costs nothing at the model level, and tokenizer extension touches only the text reasoner. We measure their effect directly instead of assuming it.

Why not just apply VGRefine at inference? It is training-free and helps, but it only redirects attention. It does not stop the language model from answering out of its priors. We treat it as additive rather than as a competitor, and the stacking experiment shows whether the two fixes target the same failure.

Proposed Experiments and Decisive Checks

All models are evaluated on the six MedVLThinker benchmarks for accuracy. On PathVQA, PMC-VQA, SLAKE, and VQA-RAD we apply the Beyond Accuracy real, blank, and shuffled protocol using full test sets, and we also run their 100-sample stratified subset for direct comparability. We report the Visual Reliance Score, Blank Drop, Image Sensitivity, and the Hallucinated Visual Reasoning Rate.

To probe the critique in Position: Reasoning After Perception Means Reasoning Without Vision, that text bridges discard spatial detail, we break results down by question type. We also evaluate localization-style items built from VGMED. We expect the bridge to be weakest on questions about where something is, and we will report this rather than average it away.

The first decisive check is whether the image-contingent reward reaches at least MedVLThinker's 54.88% average while clearly raising image sensitivity and blank drop over both MedVLThinker and the RAPID and MoCA arms. The second is whether the vocabulary conditions change bridge failure rates. Concretely, we look for an accuracy drop under the abbreviation and fragmentation perturbations, and for a positive relation between tokens-per-word of key terms and bridge failure. If neither appears, we conclude that the repair effect from the encoder study carries over to generative multimodal reasoning. That would close Gap 2 in the negative and tell practitioners to spend their effort on the reward rather than the tokenizer. The third check is whether hallucinated visual claims fall. The perceiver's findings can be compared to the image condition directly, so a findings block that survives the shuffled-image test is evidence of a description that is actually image-specific.