Normal Anatomy as a Calibration Reference for Locked Tumor-Response Models

Cancer patients on sequential multimodal therapy, meaning local treatments such as radiotherapy or surgery arranged in a planned order with drugs such as chemotherapy and immunotherapy, are scanned repeatedly over months. Models that predict response from these scans face two kinds of change between training and use. The first is an acquisition shift: a new hospital, scanner, or protocol makes the same tissue look different. The second is the treatment itself. Inflammation, edema, and necrosis caused by immune priming change how the tumor looks, and that change carries the prediction signal. Standard domain-adaptation tools treat every difference from the training data as nuisance. Applied to on-treatment scans, they risk removing exactly the change that indicates response.

We propose to correct acquisition shift using only tissue that treatment should not affect. A small, physically constrained "input adapter" sits in front of a frozen response model. It applies a smooth intensity remapping plus a slowly varying bias field, which is the kind of distortion a scanner introduces. The adapter is fit on unlabeled target-site scans so that normal reference organs far from the treatment field look, to the frozen model, the way they looked at the training site. The same correction is then applied to the whole volume, tumor included. One transform is shared across all of a patient's time points from the same scanner, so the change between the pre-treatment and mid-treatment scans is preserved. The predictor's weights never change, so performance at the original site is unchanged by construction and the device stays locked in the regulatory sense.

Why this problem, and what the prior work leaves open

The motivating evidence comes from three places. On the clinical side, the NeoRTPC02 trial of low-dose radiotherapy plus tislelizumab and chemotherapy before surgery in head and neck cancer (Liu et al.) found that several patients labeled stable disease or partial response on preoperative MRI had complete pathological response at resection. Single-cell data in the same trial point to macrophage repolarization and T-cell infiltration as the likely reason. Therapy-induced remodeling therefore looks like persistent tumor. Any model that "normalizes" on-treatment images toward untreated appearance pushes them in the wrong direction.

On the deployment side, Wu et al. report two relevant findings for a chest X-ray classifier. Only 6 of 416 FDA-cleared AI devices were retrained within two years. Fine-tuning the classifier at a new hospital recovered up to +0.23 AUC locally but cost the source site 0.176 AUC on average, and up to 0.268. Weight updates are thus both regulated and destructive.

On the transfer-learning side, The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning shows how pre-trained initialization behaves during fine-tuning. Models start with extra robustness to unseen distributions, which the paper calls effective robustness. That robustness decays toward zero as fine-tuning converges, and neither replay of source data nor L2 pull toward the pre-trained weights preserves it. This result constrains our design. It argues against trying to keep acquisition robustness inside the weights, and for doing cross-site correction outside the model.

Our method borrows its basic structure from INPUTADAPTER (Gotfrid et al.). That system trains an input-space adapter in front of a frozen clinical predictor for ICU time series, using a task loss, a fidelity loss, and a source memory bank, and reports +14.1 AUCPR for kidney-injury prediction on eICU. We differ in three ways. First, our adapter is restricted to an acquisition-shaped family of transforms, so it cannot paint lesions in or out. Second, it is fit only on reference anatomy rather than the whole input. Third, it is tied across a patient's time points. Without those constraints, an expressive adapter optimized through a frozen model is free to "explain away" on-treatment inflammation, and we expect it to do so.

Two other lines of work bear on the design. Post-hoc calibration with a learned cardiac shape prior (Ouyang et al.) handles out-of-domain MRI. Its authors note it depends on regular anatomy, which is why we put the prior on stable organs rather than on the deforming tumor. AMOS (Ji et al.) supplies the tool that makes reference tissue usable: a 15-organ abdominal benchmark across eight scanners. It also quantifies the problem, showing that segmentation drops by more than 20% mDice on MRI from unseen scanners.

Core Claim

For response prediction during sequential therapy, cross-site adaptation should be estimated on tissue whose appearance treatment does not change, and then applied everywhere. Concretely, a frozen predictor $f$ maps one or more scans to a response probability. In front of it sits an adapter $g_\phi$ that maps each voxel intensity $x(v)$ to $h_\phi(x(v))\,e^{b_\phi(v)}$. Here $h_\phi$ is a monotone intensity curve per MRI sequence, $b_\phi$ is a low-frequency bias field, and $\phi$ are the adapter parameters. The adapter is fit by

$$ \phi^\star = \arg\min_\phi \; D\big(\,\mathcal{F}_R(f, g_\phi(x_{\text{target}})),\; \mathcal{F}_R(f, x_{\text{source}})\,\big) + \lambda\,\|g_\phi - \text{id}\|^2 . $$

In this objective, $\mathcal{F}_R$ is the distribution of the frozen model's intermediate features pooled over a reference-tissue mask $R$. The mask comes from an AMOS-trained organ segmenter and is restricted to organs outside the radiation or embolization field. $D$ is a feature-statistics distance (mean and covariance matching). The second term keeps the transform close to identity, with $\lambda$ setting how strongly. One $\phi$ is fit per scanner-protocol and applied to every time point a patient has on that scanner.

The claim is falsifiable in two directions. It predicts better external-site discrimination than no adaptation. It also predicts that, unlike an unrestricted adapter, our adapter keeps the within-patient change between the pre-treatment and interim scans intact, and does especially well on cases where imaging and pathology disagree.

Research Program

The work has three parts.

First, we build the locked predictor in a way that tests the transfer-learning result in this setting. Predictors take pre-treatment and interim MRI, fused with clinical covariates. Their architecture follows the dual 3D ResNet-50 design of the breast neoadjuvant model (Multimodal deep learning models for pCR prediction). We train three versions: from random initialization, from natural-image weights, and from AMOS-pretrained 3D weights. For each, we track effective robustness across fine-tuning epochs on held-out scanners. This checks whether the decay reported for natural images replicates for 3D oncology MRI, and whether the adapter recovers what the decay removes.

Second, we build the adapter and the reference-tissue pipeline. This covers organ segmentation, exclusion of tissue inside or near the treatment field, the feature-statistics objective, and time-point tying.

Third, we build a semi-synthetic testbed where the answer is known. We insert simulated lesions into AMOS MRI and CT volumes and give them controlled "treatment" changes: a necrotic core, an edema rim, and shrinkage. We then apply acquisition shifts taken from the TorchIO corruption suite used by Ouyang et al. and from real cross-scanner pairs in AMOS. This lets us measure separately how much acquisition shift each method removes and how much treatment change it removes.

Clinical evaluation needs a multi-site longitudinal cohort with pathological response labels. No such benchmark appears in our material, so we would assemble one retrospectively from at least two sites with different scanner vendors. Candidate settings are breast neoadjuvant chemotherapy, where pCR is the endpoint, and neoadjuvant immunochemoradiotherapy in head and neck cancer, where discordance between imaging and pathology is documented.

Baselines

The naive reference is the locked predictor applied unchanged at the external site. We report its source-to-target gap, expecting a drop of the size AMOS documents for unseen MRI scanners.

The clinical-covariate model is the strongest non-imaging competitor. In the breast neoadjuvant cohort, clinical variables alone reached 0.827 AUC, against 0.888 for the full multimodal model (Multimodal deep learning models for pCR prediction), on internal validation only. Clinical variables do not suffer scanner shift, so any adapted imaging model has to exceed this model at the external site. Matching it is not enough.

Full fine-tuning on labeled target-site patients, following Wu et al., is the strongest weight-space competitor. We give it the same labeled target patients our adapter may optionally use, the same compute, and an equal hyperparameter search. We report both its target-site gain and its source-site loss; the latter was −0.176 AUC on average in their chest X-ray setting. Beating it means reaching comparable target AUC with zero source-site loss.

The strongest existing input-space method is an unrestricted INPUTADAPTER-style adapter (Gotfrid et al.) extended to 3D. It is fit on whole volumes with the same frozen predictor and unlabeled data. For 3D oncology MRI there is no reported result on Lacuna for INPUTADAPTER. Its +14.1 AUCPR on eICU kidney injury is the only figure in our material, and that is a different modality.

Two ablations isolate the key ingredients. The first is our adapter fit on whole volumes instead of reference tissue, which isolates the reference-tissue choice. The second is our adapter fit independently per time point instead of tied across them, which isolates delta preservation.

The heuristic harmonization baseline consists of bias-field correction plus histogram standardization, computed once on whole volumes and once restricted to reference tissue. Its role is explained in the next section.

Why not just…?

Why not just fine-tune the predictor at each new hospital? Wu et al. show that this trades source-site performance for target-site performance, and that cleared devices are almost never retrained. The robustness paper shows that weight-space fixes such as replay and L2 pull do not keep the robustness that pre-training provides. We still run fine-tuning as a baseline. If it reaches higher target AUC than our adapter, we will report the size of that gain against the source-site cost.

Why not just apply standard intensity harmonization? This is the most serious threat, because our adapter is a learned relative of histogram matching. Our argument is specific. Whole-volume histograms change when a tumor becomes necrotic or edematous, so whole-volume standardization partly undoes the treatment effect. Reference-tissue histogram matching does not have that flaw. If it performs as well as our feature-space adapter, the contribution reduces to the reference-tissue principle with a cheaper implementation, and we would say so. The semi-synthetic testbed decides this, because treatment change is known there and we can measure how much of it each method erases.

Why not just use an expressive input adapter, as INPUTADAPTER does? An adapter optimized through a frozen model to make target inputs "look like source" has no reason to keep post-priming inflammation. Erasing it may even lower its loss. Our prediction is that an unrestricted adapter matches ours on pooled external AUC but loses on discordant cases and on recovery of known synthetic deltas. If it does not lose there, the restriction is unnecessary.

Why not just freeze the pre-trained backbone and train a linear head? This is a plausible way to keep effective robustness. It is cheap, so we include it among the predictor variants. A frozen backbone is still exposed to acquisition shift at input, however, so it complements an input correction rather than replacing one.

Proposed Experiments

The primary clinical endpoint is external-site AUC for pathological response using pre-treatment plus interim scans, with source-site AUC reported for every method. Secondary endpoints are AUC on the subgroup that imaging labels as stable disease or partial response but pathology shows complete response, the subgroup NeoRTPC02 highlights, and expected calibration error as defined in Ouyang et al..

On the semi-synthetic testbed we report two quantities: the fraction of the simulated acquisition shift removed and the fraction of the simulated treatment change retained, measured in the frozen model's feature space and in its predictions.

On AMOS we verify that reference-organ segmentation stays reliable across scanners for the organs we use. AMOS reports weak results on adrenal glands and duodenum, so we use liver, spleen, kidneys, and muscle, and exclude any organ inside the treatment field.

The study has three decisive checks. The method works if it beats both no adaptation and the clinical-only model at the external site, loses nothing at the source site, and beats the unrestricted adapter on discordant cases. It fails if whole-volume and reference-tissue variants perform the same, which would mean treatment change was never at risk. It is made redundant if reference-tissue histogram matching ties it. Each of these outcomes is informative, and the semi-synthetic testbed makes each one measurable before any clinical cohort is assembled.