Small vision-language models fine-tuned on medical data often look more reliable than general models. When a radiologist rewords a question, they tend to give the same answer. We propose to test whether that stability is real or whether it comes from the model no longer looking at the image. The central measurement is simple. For each medical question, we ask several clinically equivalent rewordings and also swap in a different patient's scan. A model is robust in the way that matters only if its answer stays fixed across rewordings and changes when the scan changes. We then look inside the model, layer by layer, to find where each behavior is decided. Finally, we test a training objective that rewards both properties during medical fine-tuning.
This matters because two findings in the literature point in opposite directions. Text-only work shows that prompt brittleness appears when the model is unsure and its next-token distribution is flat (ProSA). Specialization should sharpen that distribution and so reduce brittleness. Yet work on discriminative medical visual question answering (VQA) finds "robustness via blindness": specialized systems keep their clean accuracy by leaning on text shortcuts, and text pathways account for 45–60% of error sensitivity (Beyond Accuracy). Nobody has measured which of these happens in generative medical assistants such as LLaVA-Med, a LLaVA model tuned on biomedical figures and instructions. Current robustness evaluations, which only check whether answers agree across prompts, cannot tell the two cases apart.
Core Claim
We hypothesize that domain specialization of a generative vision-language model (VLM) lowers paraphrase sensitivity mainly by lowering visual integration, not by making the model understand clinical language more deeply. If so, a model that is stable across paraphrases may still be ungrounded. The fix should be an objective that keeps visual integration high and even across paraphrases, not only one that makes outputs agree.
To measure visual integration we reuse the Chain-of-Embedding method of Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding. That paper compares each layer's hidden states with the image present against the same prompt run "blind", with no image. It reports that the two diverge sharply at a Visual Integration Point about 60% of the way through the network. It then sums the divergence after that point into a quantity called Total Visual Integration (TVI), which correlates with whether the answer is correct. That paper studies general benchmarks and misleading text contexts. It does not study rewordings of the same question or medically specialized models, and that is where we apply it.
Beyond the diagnostic, what is new is this. We treat grounding and paraphrase invariance as one two-dimensional target. We track how both move across the checkpoints of a specialization run. We also test whether an internal, per-paraphrase integration signal beats the output-level alternatives, which are cheaper and more obvious.
Research Program
Paraphrase and structure suite. We start from SLAKE (SLAKE), VQA-RAD (VQA-RAD) and PathVQA. For every question we build three kinds of rewording.
- Clinically equivalent lexical rewordings. These are generated by an LLM and filtered by Jaccard distance and embedding similarity, following the counterfactual-prompt recipe of FairVLM. A clinician then checks a random sample.
- Structural variants borrowed from Misaligned Roles, Misplaced Images. These move the image token after the clinical findings text or change the chat-role wrapping.
- Semantic-negation controls that should change the answer. These follow the three perturbation classes of How Multimodal LLMs Solve Image Tasks.
Counterfactual image pairs. SLAKE reuses question templates across many scans, for example asking which organ is abnormal. We can therefore pair two images that share a question but have different ground-truth answers. The image-swap flip rate on these pairs is a behavioral measure of whether the model is looking at the image. We also run a blank-image control.
Controlled specialization trajectory. Comparing off-the-shelf LLaVA-1.5 with LLaVA-Med mixes specialization with other differences between the two releases. We therefore rerun the two-stage LLaVA-Med recipe (LLaVA-Med) from LLaVA-1.5 and save intermediate checkpoints. At each checkpoint we measure four things:
- accuracy for each paraphrase and the worst accuracy across paraphrases;
- the answer flip rate across equivalent paraphrases;
- the image-swap flip rate;
- TVI and the depth of the Visual Integration Point for every paraphrase.
We also train layer-wise linear probes, as in How Multimodal LLMs Solve Image Tasks. These test whether the "lexical integration" band, where probe accuracy fell by up to 40% under rewording in LLaVA-1.5 layers 5–11, narrows, deepens, or collapses as the model specializes. To test whether the effect depends on architecture, we repeat the whole trajectory on Qwen2-VL-7B. That paper found Qwen2-VL spends its layers differently from LLaVA.
Intervention. Let $I(v,p)$ be the TVI for image $v$ and paraphrase $p$: the summed layer distance, after the integration point, between hidden states with the image and hidden states for the blind input. During the second fine-tuning stage we minimize
$$\mathcal{L} = \mathcal{L}_{\text{SFT}} + \lambda_1\,\mathrm{KL}_{p,p'} + \lambda_2\,\mathrm{Var}_{p}\big[I(v,p)\big] + \lambda_3\,\max\big(0,\ \tau - I(v,p)\big).$$
The terms are as follows.
- $\mathcal{L}_{\text{SFT}}$ is the usual next-token loss.
- $\mathrm{KL}_{p,p'}$ is the divergence between answer distributions for two paraphrases of the same question.
- The variance term asks the model to draw on the image equally no matter how the question is worded.
- The hinge term, with floor $\tau$, prevents the variance term from being satisfied by integrating nothing for every paraphrase.
The first two terms together are essentially the output-level recipe prior work already uses. The last two terms are the internal ingredient under test.
Baselines
The naive reference point is stock LLaVA-Med and our own replication of its standard second-stage tuning, evaluated with the canonical question wording. It fixes the clean-accuracy bar. LLaVA-Med reports average VQA-RAD accuracy rising from 35.23% for general LLaVA to 74.05% (LLaVA-Med). Our replication should land near that figure before any comparison counts.
The strongest existing competitor for prompt robustness is multi-paraphrase fine-tuning with an output-consistency penalty. This combines the multi-template adversarial training of Misaligned Roles, Misplaced Images with FairVLM's counterfactual prompt regularization. It is the strongest competitor because it directly optimizes the quantity robustness benchmarks score. In its original setting it drove attack success on AdvBench from 42.5% to 0% while keeping VQA-V2 performance. We found no reported result on Lacuna for paraphrase consistency of any generative VLM on SLAKE, VQA-RAD or PathVQA, so we establish that number ourselves.
The strongest existing grounding method is TVI regularization applied without paraphrases, as in Understanding Language Prior. It raised LLaVA-1.5 MME perception from 1369.75 to 1400.44.
For clean medical accuracy, the reference ceiling is R-LLaVA, which blends region-of-interest markings into the image. It reports SLAKE-EN accuracy of 89.47% on open-ended and 90.13% on closed-ended questions (R-LLaVA). It uses ground-truth boxes, so we treat it as a ceiling rather than a direct competitor.
The key ablation replaces our two internal terms with an output-level image-contrast term. This term pushes apart the answer distributions for the real image and the blank image, and needs no hidden states. It isolates whether looking inside the model buys anything.
Fairness of comparison works as follows. Our objective needs one extra blind forward pass per example. Every baseline therefore gets matched extra forward passes, spent on additional paraphrase samples. All methods share the same data, paraphrase pool and hyperparameter-search budget, and each is tuned on a held-out split of paraphrases.
Beating the baselines means two things. We must match the consistency baseline's worst-case paraphrase accuracy within noise. We must also significantly raise the image-swap flip rate on counterfactual pairs and hold clean accuracy within one point.
Why not just…?
Why not just use one or two in-context examples? ProSA found that moving from zero to one example gave the largest reduction in sensitivity. That is a genuine competitor for output agreement, so we include it at inference time. However, On the Relation between Sensitivity and Accuracy in In-Context Learning shows that the order of the examples adds a new source of brittleness. More importantly, nothing about an in-context example forces the model to use the scan. We expect its image-swap flip rate to stay low, and we measure it directly.
Why not just train on many paraphrases? This is the real threat. If it yields both high consistency and a high image-swap flip rate, our internal terms are unnecessary, and we will report that. Our hypothesis predicts otherwise. Beyond Accuracy found that black-box robustness training in discriminative medical VQA converges to ignoring the image. A consistency loss can be fully satisfied by a blind model, so nothing in it pushes against that failure. The counterfactual-pair flip rate settles the question.
Why not just measure image dependence behaviorally and regularize outputs? The image-contrast ablation is exactly this test, and it may work. Internal integration has two possible advantages. First, it is computed per paraphrase and per layer, so it can target specific wordings that switch off vision even when the final answer happens to be right. Second, the Chain-of-Embedding paper shows that output and attention signals can mislead: high attention to image tokens can coexist with no change in the representation. If the output-level ablation matches our objective, the contribution shrinks to the diagnostic and the trajectory findings, and we will say so.
Why not just use a strong general model instead of specializing? The medical-specialization study (Does medical specialization of VLMs enhance discriminative power?) found that general models with stronger text backbones sometimes beat specialized ones. For example, LLM2CLIP scored 94% versus BiomedCLIP's 91% on Brain MR. LLaVA-Med, however, shows general generative assistants performing poorly on VQA-RAD. We keep general LLaVA-1.5 and Qwen2-VL in every table so readers can see where specialization helps grounding and where it only suppresses brittleness.
Proposed Experiments
The first decisive check is the trajectory. If the specialization hypothesis holds, the paraphrase flip rate and TVI should both fall across specialization checkpoints. The image-swap flip rate should fall with them. We should also see the Visual Integration Point move deeper, or the gap between image and blind hidden states flatten, under non-canonical wordings.
If instead the paraphrase flip rate falls while TVI and the image-swap flip rate hold steady, then ProSA's confidence account is the better explanation, and the specialization hypothesis is wrong. Structural variants matter clinically because they mimic real report formats. We analyze them separately, since the Misaligned Roles paper found they shift internal features far more sharply than wording changes do.
The second check is the intervention comparison described in the Baselines section. We run it on SLAKE-EN, VQA-RAD and PathVQA. Paraphrases are held out by generation template, so the model never trains on the style it is tested on.
The third check asks whether per-paraphrase TVI, which costs one extra blind pass, predicts errors as well as SENSEL. SENSEL, the method in On the Relation between Sensitivity and Accuracy, decides when to abstain by measuring disagreement across many prompts. We score both by the area under the F1-coverage curve. SENSEL reports an average +4.1-point gain on that curve over probability baselines for text-only GPT-J-6B. Matching it at a fraction of the forward passes would give clinicians a cheap way to flag unreliable answers.
References
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Beyond Accuracy: Robustness Analysis of Encoders and Fusion in Medical Visual Question Answering
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
- Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering
- Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions
- FairVLM: Enhancing Fairness and Prompt Sensitivity in Vision Language Models for Medical Image Segmentation
- Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots
- How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
- LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
- R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis