Grounding That Looks: Penalizing Text-Only Location Priors in RL-Trained Medical Phrase Grounding

Medical phrase grounding asks a model to draw a box on a chest X-ray around the region a clinical phrase describes, such as "small left apical pneumothorax." Recent gains here come from two kinds of specialization. One is domain-pretrained language encoders. The other is reinforcement learning (RL) that rewards correct boxes. Both may be partly learning something they should not: where findings usually appear. A model that has read enough radiology text knows that "apical" means the top of the lung and "left" means the right side of the image. It can place a plausible box without examining the image. We propose to measure how much of the reported grounding progress is explained by this text-only location prior. We then change the RL reward so the policy is penalized specifically for landing on the prior in cases where the prior is wrong.

The core claim has two parts. The first is a hypothesis we will test directly: domain-specific language pre-training strengthens text-only location priors, so part of its measured grounding benefit comes from language rather than from better image–text alignment. The second is a method. RL-based grounding in the style of MedGround-R1 can be de-biased by a reward term that varies within each sampled group of answers. This detail matters because, as explained below, the obvious fix of reweighting hard examples cancels out under group-relative advantage normalization.

Background and the gap

MedRPG builds a compact grounding model: a ResNet-50 image backbone, a BERT text encoder, a small vision-language transformer, and a box-regression token. It shows that such models overfit to dataset location priors. Its attention-level contrastive regularizer (TaCo) counteracts this, and removing TaCo costs about 3.8% accuracy. MedRPG uses a general-domain BERT and supervised training only. MedGround-R1 takes a different route. It trains a vision-language model with Group Relative Policy Optimization (GRPO), an RL method that samples several answers per prompt and scores each against the group mean. Its rewards are a format reward, a binary IoU>0.5 spatial reward, and a semantic reward from a frozen MedCLIP applied to the predicted crop. It reports large mIoU gains. Nothing in its reward asks whether the box depends on the image. A policy that learns "apical → top-of-lung box" is rewarded whenever that guess happens to be right.

On the pre-training side, BERTHop found that swapping BERT for clinically pretrained BlueBERT raised OpenI AUC by 6.46%. GLoRIA pairs BioClinicalBERT with word-to-region attention and obtains large label-efficiency gains. These results motivate domain encoders. They also raise our question: when the task is localization, how much of the encoder's value is image-independent spatial knowledge carried in the words? No paper in our file reports a text-only grounding baseline on MS-CXR or ChestX-ray8, so the size of this effect is unknown.

Core mechanism

We first train a blind prior $b_0(t)$: a box predictor that sees only the phrase $t$. It is a text encoder plus a small regression head, trained on the grounding training split. It estimates what can be localized from language alone. Every test and training example can then be labeled as prior-consistent ($\mathrm{IoU}(b_0(t), b^*) \ge \tau$, where $b^*$ is the ground-truth box and $\tau=0.5$) or prior-violating.

The obvious fix is to up-weight prior-violating examples in the reward. Under GRPO this does nothing. Advantages are computed by subtracting the group mean and dividing by the group standard deviation for each prompt, so any per-prompt constant or scale factor cancels. The fix has to act at a different level. We use two ingredients that do not cancel.

The first is a within-group prior penalty. For a sampled box $\hat b$ we add

$$R_{\text{prior}}(\hat b) = -\beta \,\mathrm{IoU}(\hat b, b_0(t)) \cdot \mathbb{1}\left[\mathrm{IoU}(b_0(t), b^*) < \tau\right],$$

where $\beta$ is a weight and the indicator restricts the penalty to prompts where the prior is wrong. This term differs across samples in the same group, so it survives normalization. On such prompts it separates "confidently wrong in the stereotyped place" from "wrong elsewhere" and from "right." On prompts where the prior is correct, it leaves the MedGround-R1 reward untouched.

The second ingredient is prior-aware prompt sampling. Reweighting cannot happen inside the advantage, so we move it to the data loader and oversample prior-violating prompts in RL batches. We also construct counterfactual prompts. These are horizontally flipped images with left/right swapped in the text, plus same-finding pairs from different patients whose boxes lie in different places. They add prior-violating cases beyond the naturally occurring ones.

Domain pre-training enters in two ways. It is the independent variable of our diagnostic: we compare blind priors and full models built on general BERT against those built on BioClinicalBERT, PubMedBERT, and BlueBERT. It is also a design choice for $b_0$ itself, since a stronger domain encoder yields a stricter prior to push against.

Research Program

The work proceeds in three stages. Stage one is diagnostic. We train blind priors with each text encoder on MS-CXR, ChestX-ray8, and the 2D-adapted M3D-RefSeg. We report their accuracy and mIoU and split each test set into prior-consistent and prior-violating subsets. We then re-evaluate MedRPG-style models with general and domain encoders, and a reproduced MedGround-R1, on both subsets. The hypothesis predicts that domain encoders make the blind prior markedly stronger. It also predicts that the gain of domain encoders and of GRPO over their baselines will be concentrated in the prior-consistent subset.

Stage two adds the prior penalty and prior-aware sampling to the MedGround-R1 recipe. We keep its base model, supervised cold start, rewards, and sampling budget fixed.

Stage three runs image-dependence probes on all models. The first probe is the image-swap consistency gap. We feed the same phrase with different images of the same finding and compare how similar the predicted boxes are with how similar the true boxes are. A prior-bound model shows prediction similarity well above ground-truth similarity. The second probe is flip fidelity: whether predictions track a laterality-swapped flip. The third probe is a blank-image test, which checks whether the model still outputs a confident, stereotyped box when the image is uninformative.

Baselines

The naive reference points are a fixed center box and the blind prior $b_0(t)$ itself. If the blind prior reaches a large fraction of published accuracy, that alone is a finding. No reported result was found on Lacuna for a text-only grounding baseline on these benchmarks.

The strongest existing method is MedGround-R1. It reports mIoU 79.02 on MS-CXR (versus 73.33 for BiRD), 53.12 on ChestX-ray8, and 60.10 on 2D M3D-RefSeg (MedGround-R1). It is the right competitor because it is the current RL recipe we modify, and any gain must come on top of its semantic and spatial rewards. We will reproduce it with the same base model, cold-start data, number of samples per prompt, and RL steps as our variant, and report our reproduction next to the published figure. We also include MedRPG, which reports 69.86% accuracy at IoU>0.5 on MS-CXR (MedRPG). It represents supervised attention-level debiasing. We run it with both BERT and a domain encoder, with an equal tuning budget.

Beating MedGround-R1 means clearly higher mIoU on the prior-violating subset of each benchmark and a smaller image-swap consistency gap, with overall mIoU within noise of or above the reproduced baseline. Averaging across seeds and bootstrap intervals over test items defines noise. Three ablations isolate the ingredients: penalty only, oversampling only, and a variant that reweights advantages per prompt. The last is predicted to match the baseline, which would confirm the cancellation argument. A fourth arm adds flip augmentation with laterality swapping to MedGround-R1.

Why not just…?

Why not just add flip augmentation with left/right swapped in the text? This is a real threat for laterality errors, and it is included as a baseline arm. It cannot break vertical and anatomical priors, such as apical pneumothorax, basal effusion, or tube positions, which a flip leaves intact. If this arm closes the prior-violating gap as well as our reward does, we will report that the cheaper fix suffices.

Why not just add TaCo or reweight hard examples during supervised training and skip the RL change? TaCo already reduces location bias in supervised models (MedRPG). A policy that later receives IoU-only rewards can nonetheless relearn shortcuts, because nothing in RL penalizes them. We test this by adding TaCo-style regularization and prior-violating oversampling to the cold start only, then running standard GRPO. If the debiasing survives RL, the penalty term is unnecessary.

Why not just use a domain-pretrained encoder, since domain pre-training beats PEFT choices and generic models? BERTHop and A Comparative Study of PEFT Methods for Python Code Generation support domain pre-training for classification and generation. Our hypothesis is that for localization it partly improves the text prior rather than the image evidence. Stage one tests this directly. If domain encoders help equally on the prior-violating subset, the hypothesis is refuted and the encoder is simply the better default.

Why not just prompt a large generalist VLM? Compact specialized models have repeatedly beaten larger generalists in this literature, for example in AgriPerceiver and in DermFM-Zero (dermatology foundation model). More importantly, a generalist's grounding would face the same prior question. We will run one zero-shot generalist through the same subset split as a reference, not as the main competitor.

Decisive Checks

The central experiment is the subset-split table across MS-CXR, ChestX-ray8, and 2D M3D-RefSeg for the blind priors, MedRPG (general and domain encoder), reproduced MedGround-R1, and our variant, reported with seeds and bootstrap intervals. The proposal succeeds if it meets three conditions. The blind prior must explain a substantial share of baseline accuracy. The prior penalty must raise prior-violating mIoU without hurting prior-consistent mIoU. The image-swap consistency gap must shrink.

It fails informatively in two ways. If blind priors are weak, location shortcuts are not the bottleneck on these benchmarks. If flip augmentation or cold-start debiasing matches our variant, the cheaper fix should be preferred.

The advantage-reweighting ablation is a clean test of the cancellation argument, independent of everything else. That is why it is reported separately.