Many medical segmentation models now also read a short clinical record alongside the image: age, a blood biomarker, a line of report text. These models are often trained with a mixed loss, one term that scores each pixel independently and one term that scores overlap of the whole predicted region. The reported improvements from adding clinical data are large. In multi-center prostate cancer detection, for example, clinical prompts alone added more than 7 points of AUROC. Hospitals differ in which patients they see, though. A referral center sees sicker patients, and its biomarker distributions differ from a screening clinic's. A clinical vector can therefore tell the model which hospital the case came from, and so what that hospital's base rate of disease is, without saying anything about this particular patient. Spatial generalization research has shown repeatedly that random splits reward this kind of site memorization. The starkest case is in online road mapping, where a model's score fell from 72.7 to 26.2 mAP once test locations were kept geographically apart from training locations (Localization is All You Evaluate).
This proposal builds a measurement that splits any multimodal gain into a patient-level part and a site-level part. The test is simple. Within each hospital, shuffle the clinical vectors across patients, which keeps each site's metadata distribution intact while breaking the link to the individual. Then compare that with shuffling across hospitals. Using this measurement, we test two specific hypotheses about where site-level leakage enters. The first is that fusing metadata through a global attention pathway lets it rewrite predictions at every pixel, which amplifies site shortcuts. The second is that a loss weighted toward the region-overlap term pushes the model to use metadata for an image-level "is there any lesion here" decision, which is exactly where hospital base rates help. If both hold, the practical recommendation follows directly: route metadata through a late, low-capacity pathway and train with a loss weighted toward the per-pixel term. We then test whether that design keeps the patient-level gain when the model is evaluated on a hospital it never saw.
Core Claim
The claim has three parts. First, leave-one-site-out evaluation is necessary but not sufficient to show that clinical metadata carries patient information. A held-out hospital can still share case-mix structure with the training hospitals, so a metadata pathway that learned "site type implies prior" can still help there for the wrong reason. Second, the size of the site-level share depends on design choices that are usually treated as unrelated: where metadata enters the network, and the weights $\alpha$ (per-pixel cross-entropy) and $\beta$ (soft Dice) in the hybrid loss $L = \alpha L_{\text{CE}} + \beta L_{\text{Dice}}$. Third, a model that restricts metadata to a late gate and uses a cross-entropy-heavier loss can match the held-out-site accuracy of global-attention conditioning while relying far less on site-level signal.
We define the decomposition on a held-out site. Let $G_{\text{full}}$ be the gain over an image-only model with true metadata. Let $G_{\text{within}}$ be the gain when metadata is permuted among patients within each site and label stratum, and let $G_{\text{across}}$ be the gain when it is permuted across all sites. Then $G_{\text{full}} - G_{\text{within}}$ estimates the patient-level contribution, and $G_{\text{within}} - G_{\text{across}}$ estimates the site-level contribution. We will treat label-stratified within-site permutation carefully. Stratifying on the case-level label would leak the target, so the main analysis permutes within site only. Label-stratified permutation is reported separately as an upper bound on the gain metadata can provide by predicting lesion presence.
Grounding in Prior Work
Spatial generalization work motivates the proposal and supplies the evaluation logic. The aerial labeling benchmark showed that same-city random tiles overstate adaptability (Inria aerial benchmark). Lilja et al. quantified leakage with a spatial buffer and found large collapses (Localization is All You Evaluate). Networks have also been shown to identify the camera rather than the object (Camera Bias). The research file lists "domain confounding", meaning that disjoint splits mix spatial, sensor, and population shift together, as an open problem. Our permutation decomposition is a targeted response to that confounding for the population component specifically.
Two multimodal systems set the bar. They are the closest prior work and the setting we build on. ProstNFound freezes MedSAM and conditions it on prompts built from an ultrasound patch encoder and from age, PSA, and PSA density. Under leave-one-center-out evaluation across 5 centers it reports core-level AUROC 78.3% versus 71.1% for TRUSFormer (ProstNFound). It already does held-out-center testing. It does not separate how much of the clinical-prompt gain is site-level, and that is the question we add. LLMSeg couples a 3D residual U-Net with a frozen Llama2-7B through bidirectional cross-attention for radiotherapy target contouring (LLMSeg). Its reported breast Dice is truncated in our file, so no complete reported number was found on Lacuna for it. PECon, which aligns CT and EHR embeddings contrastively, reports AUROC 0.943 on RadFusion (PECon). Because it is a classification system, we use it only as a secondary check of whether the decomposition transfers beyond segmentation. The bone-tumor study argues that metadata such as age and lesion site is genuinely diagnostic (Hinterwimmer et al.). We take that seriously: our hypothesis is not that metadata is useless, only that its measured gain is a mixture.
Two pieces of prior work constrain the global-context direction. AMOS found well-tuned 3D CNNs beating Swin-UNETR (AMOS), and the shortcut-learning study warns that position encodings in Vision Transformers can bind spurious correlations to image coordinates (Shortcut Learning in Medical Image Segmentation). We therefore do not assume attention helps. We treat "metadata injected into a global attention pathway" as a hypothesized amplifier of shortcuts, to be tested.
For the loss, compound cross-entropy-plus-Dice objectives are standard practice, for instance for white-matter lesions under 1% of volume (Wu et al.) and COVID lung lesions (Saxena et al.). These works supply the method. The research file flags as unstudied both their behavior under domain shift and their tendency toward overconfident pixel scores. Our mechanism for the loss hypothesis runs as follows. Soft Dice is normalized per image, so on images with little or no foreground the cheapest way to lower the loss is a confident image-level presence decision. Metadata that encodes the site's base rate is well suited to making that decision.
Research Program
We build one segmentation backbone and vary two things: where metadata enters, and how the loss is weighted. The backbone is a 3D or 2D U-Net, chosen because AMOS shows tuned CNNs are competitive. Metadata, whether a tabular vector or short text, enters in one of three places. In global fusion, metadata tokens join a bottleneck self-attention block and cross-attend to all spatial positions, in the style of LLMSeg or ProstNFound. In mid-level fusion, metadata applies a per-channel feature scaling at the decoder. In late gate fusion, metadata only feeds a two-parameter adjustment of the image-level lesion-presence logit and the overall calibration temperature, and it cannot alter spatial structure. We cross each fusion design with a grid of $\beta/(\alpha+\beta)$ in $\{0, 0.25, 0.5, 0.75, 1\}$. Every configuration is trained under leave-one-site-out evaluation. On each held-out site we compute $G_{\text{full}}$, $G_{\text{within}}$, and $G_{\text{across}}$, using 50 permutation draws for confidence intervals.
The primary testbed is a multi-center segmentation dataset with case-level clinical variables. The ProstNFound five-center micro-ultrasound setting is the natural target, and we will seek access to it, since we do not know from the file whether it is public. LLMSeg's radiotherapy contouring setting is the second target. RadFusion serves as the classification check. AMOS has no clinical record but does span multiple sites, so we use it only to measure how much position-encoding shortcuts in the global block hurt held-out-site Dice independently of metadata. That separates the attention effect from the metadata effect. Metrics are held-out-site Dice, lesion-level detection AUROC (case-level for prostate cores), expected calibration error of pixel and case probabilities, and the two decomposition terms.
Baselines
The naive reference point is the image-only U-Net trained with the same loss grid and tuning budget. Every gain is measured against it, and if no fusion design beats it on held-out sites, the multimodal gain did not transfer.
The strongest existing competitor is ProstNFound-style prompt conditioning (ProstNFound): clinical variables encoded by MLPs and injected as prompts into a frozen foundation segmenter. Its reported number is core-level AUROC 78.3% under leave-one-center-out evaluation on 5 centers. Where we can access that data, we rerun it with the published configuration and also reimplement it on our backbone. Every method gets the same splits, the same number of hyperparameter trials (we fix 20 per method on inner-fold validation), and the same GPU-hours. Beating it means two things together. The late-gate model must match or exceed its held-out-center AUROC within the permutation confidence interval. It must also show a site-level share $(G_{\text{within}} - G_{\text{across}})/G_{\text{full}}$ at least halved. Matching accuracy while depending less on site signal counts as a win. Matching only the site-level share while losing accuracy does not.
The cheap fix a reviewer will propose is per-site metadata standardization, which z-scores each clinical variable within its hospital. This is a real competitor and we run it under every fusion design. Domain-adversarial removal of site information from the metadata embedding is the second correction baseline. No reported result was found on Lacuna for it in this setting, so we will report our own.
The key ablations isolate the two hypothesized amplifiers. Fusion location is varied at fixed loss weights to test attention, and the Dice weight is varied at fixed fusion to test the loss. A third ablation removes absolute position encodings from the global block, to separate positional shortcuts from metadata shortcuts.
Why not just…?
Why not just standardize metadata within each site? This removes differences in site means, and it may remove most of the site-level share. If so, that is a useful and publishable negative result, and our decomposition is what would demonstrate it. Standardization has two limitations, though. It cannot be applied to a new hospital before that hospital's statistics are collected. It also does not remove higher-order case-mix signal, such as the joint distribution of PSA and age. We measure $G_{\text{within}} - G_{\text{across}}$ after standardization. If that residual is near zero, we will recommend standardization over the late gate, and we commit to doing so.
Why not just drop the metadata? The bone-tumor evidence (Hinterwimmer et al.) and the size of ProstNFound's gain suggest that real patient information would be lost. The decomposition answers this question directly. If $G_{\text{full}} - G_{\text{within}}$ is near zero on held-out sites, dropping metadata is the correct choice, and we will say so.
Why not just rely on leave-one-site-out evaluation, as ProstNFound already does? A held-out hospital can resemble training hospitals in case mix, so leave-one-site-out evaluation reports the total transferred gain without attributing it. The comparison of within-site and across-site permutation on that same held-out site is the experiment that shows whether this concern is real. If the site-level term is negligible everywhere, leave-one-site-out evaluation was sufficient, and our contribution reduces to a confirmatory audit.
Why not just tune α and β per task as usual? Tuning on random or inner-fold validation selects whatever weighting exploits site priors best, because those priors help in validation. The loss grid under the decomposition tests whether the selected weighting is the one that leaks most.
Decisive Checks
Three outcomes would settle the proposal. The first concerns the loss: if the site-level share rises monotonically with Dice weight under global fusion and stays flat under the late gate, the loss hypothesis is supported. If it is flat across the Dice-weight grid, the hypothesis is refuted. The second concerns fusion: if global fusion shows a larger site-level share than the late gate at equal held-out accuracy, the attention hypothesis holds. The third concerns the cheap fix: if per-site standardization alone drives the site-level share to within noise of zero, the recommended design changes to standardization and the late gate is unnecessary. A secondary check on RadFusion, comparing PECon-style alignment against image-only performance and permuted-EHR controls, tests whether the decomposition generalizes from segmentation to multimodal classification.
References
- Localization is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix it
- Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark
- Camera Bias in a Fine Grained Classification Task
- ProstNFound: Integrating Foundation Models with Ultrasound Domain Knowledge and Clinical Context for Robust Prostate Cancer Detection
- LLM-driven multimodal target volume contouring in radiation oncology
- PECon: Contrastive Pretraining to Enhance Feature Alignment Between CT and EHR Data for Improved Pulmonary Embolism Diagnosis
- Impact of metadata in multimodal classification of bone tumours
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Shortcut Learning in Medical Image Segmentation
- Deeply supervised network for white matter hyperintensities segmentation with transfer learning
- COVLIAS 3.0: cloud-based quantized hybrid UNet3+ deep learning for COVID-19 lesion detection in lung computed tomography