Retrieving Different References for "What" and "Where" in Medical Visual Grounding

Vision-language models, which answer questions about images in text, are good at saying what is in a medical image and poor at saying where it is. They name a pathology but put it on the wrong side, or miss organ boundaries. One of the few methods that clearly narrows this gap shows the model an annotated reference image next to the target, so it can reason about location relative to the labeled anatomy. That method still uses one pre-selected reference per query. The natural next step is to fetch references automatically from a large annotated library. This proposal argues that the obvious way of doing so, retrieving the most semantically similar images, repeats the classification-versus-localization conflict familiar from object detection. An image that looks like "the same disease" is often the wrong guide to "the same geometry," because it may come from a different view, the opposite side, or a different scale.

We propose task-split retrieval. Semantic neighbors serve as in-context demonstrations for the naming and diagnosis part of a query. Geometric neighbors, ranked by a layout-preserving descriptor, serve as spatial references for the localization part. The agreement among several geometric neighbors gives an almost free uncertainty signal. It decides whether an expensive second refinement pass is worth running, which addresses the cost of iterative self-correction. The claim can be tested and could fail. On medical localization benchmarks, geometry-keyed references should beat semantics-keyed references at equal compute, and agreement-gated refinement should match fixed multi-pass refinement while using about half the forward passes.

Core Claim

Retrieval keys inherit the invariances of the encoder that produces them. Classification features are built to ignore where things are: deep convolutional features behave like a bag of features (Analyzing the Dependency of ConvNets on Spatial Information), and CLIP's global token absorbs diffuse context and discards edge structure (Object-level Semantic and Spatial Distillation for Open Vocabulary Detection). Localization needs features that shift when the object shifts (R-FCN).

Detection research learned to give these two tasks separate heads and separate sampling locations. Revisiting the Sibling Head in Object Detector lets each head sample its own spatial points. Disentangle Your Dense Object Detector separates receptive fields across branches. Adaptive Multi-task Learning for Few-Shot Object Detection uses separate network blocks plus a gradient balancer. Object-level Semantic and Spatial Distillation for Open Vocabulary Detection distills semantics from CLIP and boundaries from a separate spatial teacher.

All of that work decouples the two tasks inside the network. We apply the same decoupling to which examples the model is shown. Our specific prediction is that, on medical images, CLIP-nearest neighbors frequently differ from the query in laterality or view. References chosen that way should hurt localization even when they help diagnosis.

Research Program

The system has three parts.

Annotated memory bank. The bank holds images with masks, boxes, and labels, drawn from the training splits of the evaluation datasets.

Two-key retriever. A semantic key, the CLIP or VLM embedding of the image together with the question, first filters the bank to the top $M$ candidates of the right modality and structure. A geometric key then re-ranks those candidates. This key is a coarse grid of DINOv2 patch features that keeps spatial layout, and it is not flip-invariant. Re-ranking uses an optimal-transport patch-matching cost, the same correspondence tool RAU: Reference-based Anatomical Understanding with Vision Language Models uses inside its model. The top $k$ geometric neighbors become spatial references. The top semantic neighbors become text-and-image demonstrations of the answer format and diagnostic vocabulary, in the spirit of the CLIP-based selection in An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA. We keep images rather than captions, because that paper documents that captions lose spatial detail.

Gated refiner. The grounder is an RAU-style model: Qwen2.5-VL-7B fine-tuned with a reference image and emitting a segmentation token for SAM2. It runs once with the top geometric reference and produces a mask $\hat{m}_0$. In parallel, each of the $k$ references transfers its annotation to the target through the patch correspondence alone, with no model forward pass, giving masks $t_1,\dots,t_k$. We compute

$$ g = \min\Big(\tfrac{2}{k(k-1)}\sum_{i<j}\mathrm{IoU}(t_i,t_j),\; \mathrm{IoU}\big(\hat{m}_0,\ \mathrm{maj}(t_{1..k})\big)\Big), $$

where $\mathrm{maj}$ is the pixelwise majority vote. If $g \ge \tau$, we stop. Otherwise we run one refinement pass. That pass asks the model a binary verification question about the crop, following the decomposition that Can Feedback Enhance Semantic Grounding in Large Vision-Language Models? found raises feedback F1 by up to 18 points. It then regrounds with the second-ranked geometric reference.

We cap refinement at two passes. Segmentation cascades (Accurate Lung Segmentation via Network-Wise Training of Convolutional Networks; Instance-Aware Semantic Segmentation via Multi-task Network Cascades) and recurrent refiners (DeFusionNET) consistently saturate after one to three stages. In our design, extra compute goes to inputs where references disagree, not to more rounds.

What we do not adopt. We do not adopt the shortcut-and-delta refiner of iRe2f. It cut refinement parameters by 90% in dependency parsing, but porting it to VLM coordinate refinement is a separate project. Our efficiency gain comes from skipping passes, not from making each pass cheaper. The two approaches could later be combined.

The analysis that motivates everything. Before building the system, we measure how well semantic similarity predicts geometric similarity between memory-bank pairs. We also measure how often CLIP top-1 neighbors have the opposite laterality from the query. Both measurements use annotated left and right structures in RAOS and AMOS2022-MRI. If CLIP neighbors already match laterality and view, the core claim is weak, and we will report that.

Baselines

The naive reference point is the fine-tuned grounder with no reference image. On RAOS, the baseline VQA accuracy RAU reports is 16.6%, compared with 70.7% for RAU itself (RAU). We rerun this setting ourselves rather than relying on the reported figure.

The strongest existing method is RAU with its pre-selected single reference. It reports 70.7% VQA accuracy on RAOS, Dice 0.67 on Arcade against 0.19 for fine-tuned SAM2, and 95.4% zero-shot labeling accuracy on CAMUS. It is the strongest competitor because it already contains the reference mechanism, correspondence matching, and the SAM2 decoder. Any gain we report must therefore come from which reference is chosen and when refinement happens.

The baseline that isolates our key ingredient is RAU with CLIP top-1 retrieval from the same memory bank. This is the obvious combination and our main ablation. All retrieval variants share the identical grounder weights and memory bank, so the comparison measures only the reference choice. We also include random retrieval from the same bank as a floor.

For refinement, we compare against a fixed three-pass Agent-Verifier loop in the style of Can Feedback Enhance Semantic Grounding in Large Vision-Language Models?, using the same binary verification prompt. We also compare against "always refine once," which has the same pass cap as our method but no gate.

Compute is counted in VLM forward passes and wall-clock seconds per image on the same GPU. Correspondence transfers are timed and included in our method's cost. Each method gets the same hyperparameter budget. For ours, this means searching $k$, $M$, and $\tau$ on a validation split only.

Beating the strongest baseline means two things. First, a statistically significant Dice and accuracy gain over CLIP-retrieved RAU, which should be largest on lateralized structures and on out-of-distribution CAMUS and LERA. Second, matching the three-pass loop's accuracy within noise while using at most about 1.5 passes on average.

Why not just…?

Why not just retrieve by CLIP similarity, as multimodal in-context learning already does? This is the default, and PICa shows it helps knowledge-heavy questions (An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA). The detection literature, however, says features tuned for semantics degrade localization (Object-level Semantic and Spatial Distillation for Open Vocabulary Detection). Our first experiment settles the question directly. If CLIP retrieval equals geometric retrieval on localization, the core claim fails, and we will say so.

Why not skip the language model and transfer labels from geometrically retrieved references directly? This is a real threat. Fusing labels from several registered references is a long-standing approach in medical segmentation, and our correspondence-only masks $t_i$ are exactly such a baseline. We report it as a full method. We expect it to compete on in-distribution organ masks and to fall behind where the query requires reasoning: choosing which of several structures is meant, lateral relations, and out-of-distribution modalities such as CAMUS, where RAU's model-based reasoning reached 95.4% labeling. If label transfer matches the VLM everywhere, the honest conclusion is that the retrieval key matters and the language model does not. That would still be a useful result.

Why not just scale the model or prime the prompt? U2-BENCH finds that scaling improves classification but plateaus on keypoint and organ localization. It also finds that naming the anatomical region in the prompt adds 7.3 points (U2-BENCH). We include region priming in every condition, so any gain we report comes on top of it.

Why not run a fixed number of refinement passes? The saturation results cited above suggest that later passes add cost without gain. Can Feedback Enhance Semantic Grounding in Large Vision-Language Models? itself notes the lack of uncertainty-triggered refinement. The accuracy-versus-passes curve described below tests whether our gate beats a fixed schedule.

Proposed Experiments

Datasets. We evaluate on RAOS, Arcade, and the RAU out-of-distribution sets (CAMUS, LERA, AMOS2022-MRI), plus the detection tasks in U2-BENCH: lesion localization, organ detection, and keypoint detection. On U2-BENCH the best open-source U2-Score is 0.2630 (DeepSeek-VL2) and the best closed-source score is 0.5835 (Dolphin-V1) (U2-BENCH). The file reports no RAU-style reference method on U2-BENCH, so our numbers there will be the first.

Metrics. We measure Dice, VQA and labeling accuracy, U2-BENCH localization scores, laterality error rate, VLM passes per image, and latency.

Decisive checks. There are three. First, the geometric-versus-semantic key comparison at fixed $k$, broken down by lateralized versus midline structures. Second, a horizontal-flip test on the queries: a geometric key should retrieve correspondingly flipped references, while a near flip-invariant semantic key should not. Third, a curve of accuracy against average VLM passes as $\tau$ varies, compared with the fixed one-, two-, and three-pass schedules. The gate succeeds if its curve lies above theirs.

We also check that the agreement score $g$ predicts the error of $\hat{m}_0$, measured by the rank correlation between $g$ and Dice on held-out data. Without that, the gate is just a random skip.