Chemists solved a version of medicine's scanner-drift problem long ago. A mass spectrometer's raw signal for a molecule changes from run to run, so analysts add an internal standard: a reference compound that goes through exactly the same sample and instrument. They report the ratio of the target signal to the reference signal. Any distortion that hits both equally cancels. Imaging models that try to replace needle biopsies ("virtual biopsies") never do this. They read absolute features from a lesion and mix in patient metadata such as age. As a result they drift across scanners and sites. They can also produce confident answers that come from demographics or scanner fingerprints rather than from the lesion itself, which is the classifier version of a hallucination.
We propose treating the patient's own healthy tissue in the same scan as the internal standard. The key step is to pass the reference tissue through the entire multimodal model, with the same clinical metadata attached, and not only through intensity preprocessing. The diagnosis is then read from the difference between the model's lesion score and its reference-tissue score. Anything the two inputs share cancels: the scanner, the contrast timing, the patient's physiology, and every demographic main effect. Borrowing a second idea from analytical chemistry, we score pairs of two healthy regions as "blanks." The spread of these blank scores sets a per-site limit of detection, and the model abstains when the lesion signal does not rise above that noise floor.
Core Claim
A multimodal virtual biopsy that outputs a lesion-minus-reference contrast, with the reference drawn from the same scan and conditioned on the same metadata, should do three things. It should hold its discrimination under leave-one-site-out testing better than intensity harmonization. It should remove metadata-driven shortcuts of the kind that collapsed foundation models to near-chance performance on young patients. It should give a principled abstention rule whose error rate is set by measured blank variability rather than by an arbitrary confidence threshold.
Write $s(x, c)$ for the model's logit, where $x$ is an image region and $c$ is the clinical metadata vector. Let $x_L$ be the lesion region and $x_R$ an anatomically matched healthy region of the same study. The reported evidence is
$$\Delta = s(x_L, c) - s(x_R, c).$$
If the logit decomposes as a lesion term plus a nuisance term $g(\text{site}, \text{protocol}, c)$, the nuisance term cancels exactly. This is the imaging analogue of $S_A'/S_{IS}' = S_A/S_{IS}$ in log space. Metadata can then influence $\Delta$ only by changing how much the lesion differs from the reference. That interaction is the "valid context" role that fusion papers claim for metadata. The additive shortcut role is removed.
For abstention, take two disjoint healthy regions $x_{R_1}, x_{R_2}$ from the same scan. Their blank contrast is $b = s(x_{R_1}, c) - s(x_{R_2}, c)$. The per-site threshold is
$$\tau_{\text{site}} = \mu_b + 3\sigma_b,$$
where $\mu_b$ and $\sigma_b$ are the mean and standard deviation of blank contrasts at that site. The factor of three is the conventional chemical limit of detection. We also report a conformal quantile version for exact coverage. Predictions with $|\Delta| < \tau_{\text{site}}$ are withheld as "not distinguishable from background."
Why this is not already done
The literature file shows the pieces but never this assembly.
Virtual biopsy work demonstrates the clinical need but reads lesions in absolute terms. The renal CT system in Artificial intelligence links CT images to pathologic features and survival outcomes of renal masses fuses three CT phases with cross-attention and automatic nnU-Net masks. The gastric virtual biopsy in Multimodal Artificial Intelligence‐Based Virtual Biopsy for Diagnosing Abdominal Lavage Cytology‐Positive Gastric Cancer combines radiomics with blood markers. Its AUC falls from 0.883 to 0.823 between external cohorts. The IPMN stacking model in Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification smooths decisions in probability space. None of the three uses same-patient reference tissue as a denominator. These papers motivate the proposal.
The fusion literature supplies a tension that constrains our design. JI-ADF and the retinal nephropathy transformer Trans-MUF report that adding metadata helps. By contrast, A Fairness Audit of Medical Imaging Foundation Models on a Multimodal Structured Clinical Benchmark shows metadata-driven shortcuts pushing AUROC to about 0.508 for patients aged 18–40 on INSPECT. Our contrast is a structural answer to "when is metadata context and when is it a shortcut": main effects cancel, and only interactions survive.
The closest method is IRIS. IRIS suppresses vision-language hallucinations with a ratio of $P(\text{token}\mid\text{image},\text{text})$ to $P(\text{token}\mid\text{text})$, which cancels the language prior. Our denominator differs in kind. IRIS removes the image, so its denominator input is unlike anything the model saw in training. We substitute a real, in-distribution region that shares every nuisance factor except the pathology. A null-image denominator cancels the language prior but not scanner or demographic effects. A same-scan tissue denominator cancels all of them. The analytical chemistry papers supply both the method and its known failure mode. Improved analytical performance of plasma amyloid β assay using immunoprecipitation‐mass spectrometry and the SIMS work of Hedenstierna et al. use an endogenous matrix element as the reference. The file's own list of open problems warns that the reference must experience the same distortion ($\alpha_A = \alpha_{IS}$), and that endogenous references drift during systemic disease. We test both failure modes directly rather than assuming them away.
Research Program
The model. We build a Siamese multimodal scorer: a shared image encoder, the same clinical-metadata pathway on both branches, and a head that outputs $s(x,c)$. It is trained on $\Delta$ with the diagnostic label. A second loss drives blank contrasts toward zero, so the model learns to be invariant to healthy-versus-healthy differences.
Reference regions. These are chosen by anatomy and segmented automatically with nnU-Net, following the renal CT paper. In pancreatic MRI we use the cyst-free pancreatic sub-regions that Cyst-X already annotates (head, body, tail), with liver and paraspinal muscle as secondary references. In renal CT we use normal ipsilateral and contralateral parenchyma. In CT pulmonary angiography, where emboli locations are unknown, we use slabs outside the pulmonary arteries, such as upper abdomen and thoracic wall. These carry scanner, protocol and patient identity but no embolus signal.
The language arm. We extend the same contrast to a vision-language model. A candidate finding sentence is scored by the log-likelihood difference between conditioning on the lesion crop and conditioning on a pathology-free crop of the same image. Sentences below the blank-derived threshold are flagged as ungrounded.
Analyses beyond accuracy. The central diagnostic asks whether cancellation actually holds. We fit site and protocol effects separately to lesion embeddings and to reference embeddings, and measure how much of the site variance in $s(x_L,c)$ is explained by $s(x_R,c)$. This is the imaging version of checking $\alpha_A = \alpha_{IS}$.
We also stratify by systemic disease markers where the data record them, for example eGFR in renal cohorts. The point is to see whether reference tissue itself drifts. If it does, we replace the plain difference with a calibration curve fitted on blanks, analogous to the multi-point response curves of Gao et al..
Finally, the contrast discards legitimate prior information: older age truly raises malignancy base rates. We therefore report the image-grounded evidence $\Delta$ and a separately audited metadata prior as two numbers, and test whether recombining them afterwards recovers full-model AUC without re-introducing the age gap.
Baselines
Naive reference points. The first is a single-branch model on the lesion alone, with metadata concatenated to image features, trained with the same backbone and budget. This is the "do nothing" point. The second is the same model with the metadata removed, which shows how much metadata contributes at all.
Strongest existing methods, per benchmark.
On Cyst-X, the competitor is the cUPMI stacking ensemble of Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification. It reports binary AUC 0.839 and QWK 0.595 (quadratic weighted kappa, an agreement score for ordered risk classes). It is the strongest because it is the only model in the file tuned to this cohort's multi-stream MRI.
On INSPECT, there are two reference points from A Fairness Audit of Medical Imaging Foundation Models on a Multimodal Structured Clinical Benchmark. The first is its task-specific CT-LRCN at AUROC 0.721. The second is its adversarial debiasing of MedImageInsight. That debiasing cuts the underdiagnosis-rate age gap from 0.333 to 0.069 at an overall AUROC cost of 0.011, and it is the strongest competitor on the shortcut question.
On renal CT we re-implement the multi-phase cross-attention late-fusion model of Artificial intelligence links CT images to pathologic features and survival outcomes of renal masses on the public TCIA subset and on MMIST-ccRCC. The paper reports prospective malignancy AUC 0.871 on its private cohort. No reported result was found on Lacuna for the TCIA subset alone, so our re-implementation is the comparison. On MMIST-ccRCC we also run MedMIX, which reports only relative gains of 1.8–9.1%. No absolute number was found on Lacuna for MMIST-ccRCC, so it is re-run.
On HEAL-MedVQA the competitor is LobA, reported as up to 5.44% accuracy gain. We also run IRIS-style null-image contrast on the same base model.
Fairness of comparison. All methods share data splits, backbone family and segmentation masks. Each gets the same hyperparameter search budget, set to the larger of the two methods' default searches. Every comparison is run under leave-one-site-out. For Cyst-X that means Northwestern, NYU and Mayo in turn.
What counts as beating a baseline. On Cyst-X, a held-out-site AUC above cUPMI's under the identical protocol. On INSPECT, an age gap at or below 0.069 with an AUROC cost at or below 0.011, obtained without an adversary. On the abstention curves, higher accuracy at matched coverage, which beats simply matching cUPMI's in-distribution numbers.
Ablations isolating the key ingredient. The key ingredient is that the denominator is same-scan, same-patient, same-metadata tissue. Three ablations replace it. The first uses a healthy region from a different patient at the same site. It keeps scanner cancellation but loses patient and metadata cancellation. The second uses a blank or null image, which is the IRIS-style denominator. The third applies a post-hoc contrast to a lesion-only model with no blank loss. If the different-patient ablation matches the full method, the "same patient" claim is false.
Why not just…?
Why not just divide lesion intensity by aorta or parenchyma intensity as preprocessing? Radiologists already use such ratios, and this is the most serious threat on the scanner-drift question. It enters as an arm run at equal budget: lesion-to-reference intensity normalization feeding the single-branch model. The mechanism difference is that preprocessing cancels only a multiplicative intensity factor. It cannot cancel what the network learns from texture, noise pattern, slice thickness or metadata, and the INSPECT shortcut is invisible to it by construction. If the preprocessing arm matches our method on leave-one-site-out AUC, we concede the drift claim. The proposal then rests on the shortcut-removal and detection-limit results, and we report it that way.
Why not just use adversarial debiasing, or drop the metadata? The fairness audit shows debiasing works well, reducing the age gap by 79% for 0.011 AUROC. It is therefore a real competitor on the demographic question. Its limitations are that it needs the protected attribute named in advance and removes only that attribute. Our contrast removes all additive metadata and scanner effects without naming them, and it also yields abstention. Dropping metadata altogether forfeits the interaction benefits that JI-ADF and Trans-MUF report. The INSPECT experiment settles the question directly. We also test for an unnamed shortcut, site identity, which the adversary was not targeted at.
Why not just threshold confidence or use a null-image contrast for abstention? Selective prediction by confidence, as in Mohri et al., inherits the model's own miscalibration under site shift. The IRIS null-image ratio cancels the language prior but not shared nuisance factors. Both are run as abstention baselines. The deciding measure is accuracy at matched coverage on the held-out site, where a confidence threshold tuned on source sites is expected to lose its intended error rate. A blank-derived threshold is recomputed from the new site's own healthy tissue and needs no labels.
Why not just harmonize features with histogram matching or batch-effect correction? These correct whole-volume or whole-cohort statistics. The chemistry evidence cited in the file indicates that distortions are local, which is why a co-located reference is needed. Histogram matching is included as a further equal-budget preprocessing arm.
Proposed Experiments
Experiment 1: leave-one-site-out drift test. This runs on Cyst-X across its three centers and on renal CT (TCIA subset plus MMIST-ccRCC). We report AUC, QWK and expected calibration error on the held-out site for every baseline and ablation. We also run the cancellation diagnostic: the fraction of site variance in lesion scores explained by reference scores. The method's premise fails if that fraction is low. In that case the lesion and reference do not share distortion, and a gain on the drift test would need a different explanation.
Experiment 2: INSPECT shortcut test. We report AUROC by age band and the underdiagnosis-rate age gap. As a stress test we shuffle age across patients at test time. A contrast model should be nearly invariant to the shuffle. A concatenation model should not.
Experiment 3: detection-limit abstention. We plot risk against coverage and check that the blank-derived threshold controls the false-positive rate at the unseen site near its nominal level. We compare against confidence thresholding and null-image contrast.
Experiment 4: language arm. This runs on HEAL-MedVQA, using its pathology masks to pick pathology-free reference crops from the same radiograph. It is further stress-tested with the Med-HallMark counterfactual prompts from Detecting and Evaluating Medical Hallucinations in Large Vision Language Models. We report the area under the curve for detecting hallucinated answers, and accuracy under the benchmark's textual and visual perturbation tests (TPT and VPT). The comparisons are LobA and IRIS-style contrast.
What would decide the question. The claim is confirmed if the same-patient contrast beats the different-patient ablation on held-out sites. It is further confirmed if it matches adversarial debiasing on the age gap without being told about age. The claim is falsified if reference-intensity preprocessing ties it on drift and debiasing ties it on shortcuts. In that case the internal-standard framing adds nothing beyond known tools, and we would say so.
References
- Artificial intelligence links CT images to pathologic features and survival outcomes of renal masses
- Multimodal Artificial Intelligence‐Based Virtual Biopsy for Diagnosing Abdominal Lavage Cytology‐Positive Gastric Cancer
- Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification
- JI-ADF: Joint-Individual Learning with Adaptive Decision Fusion for Multimodal Skin Lesion Classification
- A multimodal transformer system for noninvasive diabetic nephropathy diagnosis via retinal imaging
- A Fairness Audit of Medical Imaging Foundation Models on a Multimodal Structured Clinical Benchmark
- IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination
- Improved analytical performance of plasma amyloid β assay using immunoprecipitation‐mass spectrometry
- Secondary Ion Microanalysis of Fluorine in Apatites of Biological Interest
- High-Throughput Large-Scale Targeted Proteomics Assays for Quantifying Pathway Proteins in Pseudomonas putida KT2440
- MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis
- Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMs
- Learning to Reject with a Fixed Predictor: Application to Decontextualization
- Detecting and Evaluating Medical Hallucinations in Large Vision Language Models