Mode-Wise Input Prompts for Frozen Medical Segmenters

A segmentation network trained at one hospital often fails at another. The anatomy is the same, but the scanner adds a different smooth brightness drift, noise texture and sharpness. In regulated deployment the network cannot be retrained, so the only option is to edit the incoming image. Today's best tool for this, Fourier visual prompting, learns a small additive correction to the lowest Fourier frequencies of each target image. That works partly, but a Fourier coefficient is a sine wave spanning the whole image. A correction at one frequency therefore spreads across the entire slice and cannot fix a brightness drop confined to one coil region.

We propose a different correction. First, split each target image into a few oscillatory components whose frequency bands are fitted to that image. This is called adaptive signal decomposition: the data choose their own components rather than a fixed filter bank choosing them. Then learn a smooth, low-resolution gain map for each component plus a correction for the slowly varying residual. Each scale gets its own spatially varying adjustment, and the frozen network sees the recombined image. The decomposition is computed once per image and has no learnable parts. All learning happens in roughly 2,000 gain-map parameters, trained without target labels by the self-training recipe of FVP. This sidesteps the usual objection that adaptive decomposition cannot be backpropagated through.

Core Claim

At a parameter budget matched to Fourier prompting, a prompt that applies spatially smooth gains to image-adaptive modes will recover more Dice on cross-scanner MRI shift than a global Fourier prompt. It will also distort anatomical boundaries less, measured by surface distance. The gain should come from two separable ingredients. Spatial gain maps handle localized bias fields. Adaptive per-image band placement handles vendor differences in resolution and noise spectrum. The second ingredient is the part a skeptic can most easily doubt, and the experiments are built to test it in isolation.

Relation to Prior Work

Three papers supply the motivation and the closest competitors.

FVP learns an additive prompt on a 2,048-coefficient central Fourier patch, keeping the segmenter frozen and requiring neither source data nor target labels. We keep its constraints, budget and self-training, and change only the space the prompt lives in.

FreeSDG splits images with fixed Gaussian filters into low frequencies (treated as "style") and high frequencies (treated as "structure") to augment single-source training. FedDG interpolates global Fourier amplitude spectra across sites. Both retrain the model, and both use bases fixed in advance. FreeSDG's 0.795 versus FedDG's 0.745 Dice on LES-AV is mild evidence that how the spectrum is split matters. It is not evidence for adaptive splitting specifically.

INPUTADAPTER shows that a learned input-space translator in front of a frozen predictor can recover large gains on clinical time series, for example +14.1 AUCPR on MIMIC-IV to eICU acute kidney injury. It does so with a black-box network that needs labeled target data and was never applied to images. Our adapter is label-free, low-dimensional and inspectable by construction.

IMFACT supplies the conceptual mechanism: substituting individual intrinsic modes moves a classifier's decision while preserving signal morphology. It works only on 1D signals and flags mode mixing (one component absorbing several unrelated scales) as unresolved.

The decomposition itself is borrowed rather than invented. We use 2D variational mode decomposition, which fits K band-limited modes with adaptive centre frequencies by alternating optimization. It has already been used for ultrasound denoising (2D-VMD ultrasound). We choose it over bidimensional EMD (BMEMD) because BEMD's 2D spline envelopes are expensive and unstable at image borders. We use Total-Variation Mode Decomposition, with its released software, as a second adaptive decomposition. This tests whether conclusions depend on one algorithm.

Two benchmark papers set the problem size. AMOS reports drops of over 20% mDice on MRI from unseen scanners. M&Ms shows that intensity augmentation during training is the strongest known fix, cutting the unseen-vendor drop from about 6% to 1.5%. It constrains us: if a team can retrain, augmentation is a hard bar.

Research Program

For a target slice $x$, compute modes $u_1,\dots,u_K$ and residual $r$ with $x=\sum_k u_k + r$. Edges are handled by mirror padding. To make mode $k$ mean roughly the same scale across vendors, we initialize centre frequencies from source-domain averages that are stored once. The prompted image is

$$x' = \sum_{k=1}^{K} \big(1+G_k\big)\odot u_k \;+\; r + B,$$

where $G_k$ is a gain map for mode $k$ and $B$ is an additive correction to the slow residual. Both are bicubically upsampled from coarse grids, for instance $K=4$ modes plus a residual on $20\times20$ grids, or about 2,000 parameters. The symbol $\odot$ is pixelwise multiplication. A single prompt is shared across all target images from a site, as in FVP. It is optimized through the frozen network using FVP's prototype-filtered, double-threshold pseudo-labels, plus a small penalty on the roughness of $G_k$ and $B$.

The work has three parts. The first is implementation and runtime profiling. 2D VMD runs in the FFT domain and is precomputed, so inference adds only K upsampled multiplications. The second is a mode-correspondence study: do centre frequencies and mode energies stay aligned across vendors, and how often does mode mixing occur? This is measured by spectral overlap between adjacent modes. The third is the controlled and real-shift experiments below.

Baselines

The naive reference is the frozen source model applied directly to target images. The classical physics reference is the same model applied after standard MRI bias-field correction with a fixed histogram standardization to stored source statistics. This costs nothing to learn and directly targets the smooth drift we claim to handle.

The strongest existing method under the same constraints is FVP, reimplemented on our backbones. It gets the same pseudo-labeling code, the same 2,048-parameter budget, the same number of optimization steps and the same hyperparameter search budget, about 20 configurations tuned on a held-out source-like validation split. Its reported reference is Dice improving from 0.517 to 0.733 on abdominal CT to MRI (FVP). We will first reproduce this setting. Beating FVP means a mean Dice gain larger than seed-to-seed variability, measured over at least three seeds, on the cross-scanner MRI targets. It also means no worse average surface distance.

A higher-capacity reference is a small convolutional input adapter in the spirit of INPUTADAPTER, trained with the same unlabeled self-training. It shows what unconstrained capacity buys. For M&Ms we also report the retrained nnU-Net with intensity augmentation as an upper reference, not a competitor, because it violates the frozen-weights constraint. Its reported unseen-vendor gap is 1.5% (M&Ms).

The ablations are the scientific core. The first keeps spatial gain maps but replaces adaptive modes with fixed bands: stationary wavelet sub-bands, or fixed Gaussian bands as in FreeSDG. This isolates the value of adaptivity. The second keeps adaptive modes but uses scalar gains instead of maps, which isolates spatial localization. The third swaps VMD for TV mode decomposition.

Why not just…?

Why not just use intensity augmentation or histogram matching? If retraining were allowed, M&Ms shows this is very strong. Our setting forbids retraining, so augmentation is unavailable at deployment. Histogram matching to stored source statistics is available, and we include it combined with bias-field correction. It is global per intensity level, so it cannot undo a spatially varying gain. The synthetic-shift experiment measures exactly how much of the gap it closes.

Why not just prompt in a fixed wavelet basis with spatial gain maps? This is the real threat. Wavelets are localized and cheap, and if the benefit comes from spatial gain maps rather than adaptive bands, they will match us. We do not argue around this. Comparing fixed-band and adaptive-mode prompts at matched budget settles it. If fixed bands tie, we will report that the useful idea is "localized multi-scale prompting," and that adaptive decomposition adds cost without benefit. Adaptivity should matter most where vendors differ in effective resolution and noise spectrum, such as 1.5T versus 3T or different reconstruction kernels, because fixed dyadic bands then split noise and anatomy at the wrong place.

Why not just use a learned convolutional adapter? It has more capacity. Without target labels, however, a free-form image-to-image adapter trained only on pseudo-labels can hallucinate or erase structures in ways that please the frozen network. INPUTADAPTER relied on labeled target data, and FVP's low-dimensional prompt was motivated by exactly this risk. We run this adapter and report both Dice and an image-fidelity check: the change in boundary sharpness and the structural similarity of $x'$ to $x$. If it wins on both, the low-dimensional prompt is not justified.

Proposed Experiments

The controlled experiment comes first. On source-domain M&Ms and prostate MRI, with labels hidden, we inject known shifts. These are smooth random bias fields, vendor-like changes to noise spectra, and point-spread blurs, each applied separately and in combination. Every adapter is then measured on how much of the known Dice loss it recovers. Prediction: Fourier prompts recover the global-contrast component but not localized bias fields. Fixed-band and adaptive gain-map prompts both recover bias fields. Only adaptive modes should hold up when the shift is a change in resolution.

The real-shift experiments use, in order, M&Ms with the Canon vendor as the unseen target; the six-site prostate MRI set from FedDG in leave-one-site-out form, each held-out site treated as source-free; AMOS MRI from unseen scanners; and FVP's abdominal CT to MRI for direct comparability. We report Dice per structure, average surface distance, a boundary-band Dice computed within a few pixels of ground-truth contours to expose ringing, and runtime.

The decisive outcomes come from these checks. If adaptive modes beat fixed bands on real cross-scanner MRI and on the resolution-shift synthetic condition, the claim stands. If mode-correspondence analysis shows centre frequencies wandering across vendors, the shared prompt is ill-posed, and per-image alignment becomes the next problem. If FVP matches us on AMOS MRI, cross-scanner MRI shift is mostly global, and the localization argument fails for that benchmark.