Cleared medical AI models almost never change after approval. Of 416 FDA-cleared AI devices, only 6 were retrained on new data within two years (Regulating AI Adaptation). Yet the same study shows a pneumothorax classifier losing up to 0.18 AUC at a new hospital. Fine-tuning on the new site recovers that loss but costs an average of 0.176 AUC on the original site. Once the weights are frozen, the only thing left to change is the image. We propose a small front end placed before the frozen model that re-expresses each incoming scan in the "visual dialect" of the training scanner. It does this in closed form, with no gradient steps and no labels from the new site.
The front end is an analysis sparse coder. Such a coder applies a bank of learned filters to the image, so that clean anatomy gives a few large responses and scanner noise gives many small ones. It then zeroes the small responses with a threshold and resynthesizes the image. We train these filters once, on source-site images only, and record how strongly each filter responds on source scans. At a new site, for each scan, we recompute two numbers per filter from that scan alone. The first is a noise-aware threshold. The second is a gain that pulls the filter's signal response back to its source level. The frozen model then sees the resynthesized image.
In plain terms, this is intensity histogram matching generalized to many learned frequency and texture channels. That lets it correct differences in sharpness, noise floor and reconstruction kernel that plain intensity matching cannot. The whole per-scan adjustment is a short vector of bounded numbers that can be logged and audited.
Core Claim
The claim has two parts. First, in the coefficient domain of a source-trained analysis operator, a large share of cross-scanner shift reduces to per-filter changes in scale and noise floor. Second, correcting those changes analytically recovers much of the lost accuracy of a locked segmenter or classifier. Neither part holds for pixel intensities alone.
The specific prediction that makes this falsifiable is where the gains appear. Relative to plain histogram matching, gains should concentrate on site pairs whose shift lies in spatial-frequency content, such as reconstruction kernels, dose, blur and coil noise. Gains should vanish on site pairs whose shift is purely a global contrast remapping. If histogram matching ties us everywhere, the idea is wrong.
Relation to Prior Work
The mechanism is borrowed from Learning Deep Analysis Dictionaries for Image Super-Resolution (DeepAM). DeepAM stacks analysis-filter and soft-threshold layers. Because its thresholds are tied to noise variance, it adapts to unseen noise levels by rescaling the first-layer threshold, with no retraining. That result is shown only for additive white Gaussian noise on natural-image super-resolution. We extend it in three ways: to per-filter gain as well as threshold, to non-additive medical shifts, and to use as a harmonizer in front of someone else's frozen network.
The encoder design follows the projective dictionary pair line (Projective dictionary pair learning; PADDLE) and cosparse operator learning (The Cosparse Analysis Model and Algorithms; Separable Cosparse Analysis Operator Learning). These works establish that encoding is one filtering pass, with no iterative sparse solve. The resulting speedups over synthesis coding are reported at 1,000x–80,000x. We use them for their speed and not for their discriminative objectives.
Analysis-Synthesis Dictionary Learning for Universality-Particularity Representation splits dictionaries into shared and class-specific parts. That is conceptually close to separating anatomy from scanner style, but it operates across semantic classes within one domain and requires training on all parts. We instead adapt only scalar statistics at test time. Convolutional variants (CDPL-Net) put pair layers inside the network. That is exactly what a locked device forbids.
Two papers motivate the setting.
- Adapt the Data, Not the Model (INPUTADAPTER) trains an attention-based adapter in front of a frozen ICU time-series predictor. It reports +14.1 AUCPR on eICU acute kidney injury prediction. However, it needs labeled target data and a trained network adapter. We need neither, and we work on 2D and 3D images.
- TASD is the closest medical-imaging competitor. It learns a dictionary of source shape masks and updates batch-normalization parameters at test time by gradient descent. Its dictionary lives in label space, and it changes model parameters. Ours lives in image space and leaves the model untouched.
Two findings constrain the design.
- AMOS reports a drop of more than 20% mDice on unseen MRI scanners even for strong 3D models. Architecture alone does not remove the gap.
- DeepAM's threshold rule is derived for white Gaussian noise, and bias fields are spatially varying and multiplicative. We therefore test global against locally tiled statistics, and a log-intensity variant.
Research Program
We learn a convolutional analysis operator $\Omega$ with $m$ filters and a synthesis operator $D$ on source images only. Training minimizes reconstruction error of $x \approx D\,S_\lambda(\Omega x)$, plus an $\ell_1$ penalty on the coefficients and a near-tight-frame penalty that keeps $D\Omega$ close to identity. Here $S_\lambda$ is element-wise soft thresholding with per-filter thresholds $\lambda$. One filter is constrained to be low-pass, so that global contrast is carried by a single channel. This makes plain histogram matching a special case of our method.
An optional second training term asks the frozen predictor to give the same output on $x$ and on its reconstruction. The model's weights still never change. On source data we store, for each filter $j$, a robust signal scale $\sigma_j$ (median absolute coefficient above the noise floor) and a noise scale $\nu_j$.
At test time, for a single target scan $x_t$ with coefficients $c_j = \Omega_j x_t$, we estimate a noise scale $\hat\nu_j$ from the median absolute deviation of the finest-scale coefficients. We then set the threshold $\hat\lambda_j = \lambda_j \hat\nu_j / \nu_j$, following DeepAM's variance-tied rule. Next we estimate the post-threshold signal scale $\hat\sigma_j$ of the scan. The harmonized image is
$$\hat x_t = D\,\big(g \odot S_{\hat\lambda}(\Omega x_t)\big), \qquad g_j = \sigma_j / \hat\sigma_j,$$
where $g$ is the vector of per-filter gains and $\odot$ is element-wise multiplication. Gains are clipped to a preset range. The frozen model receives $\hat x_t$. Everything is computed from the scan itself in one forward pass, and the adjustment is a vector of $2m$ numbers.
A variant replaces the global scale $\hat\sigma_j$ with a scale computed per spatial tile, which targets bias fields. A second variant matches full per-filter quantile functions instead of scale alone.
The main risk is content confounding. A different field of view or organ size also changes filter responses, and matching those changes would distort anatomy. We address this by computing statistics inside a coarse body or foreground mask, and by measuring directly whether harmonization degrades performance on held-out source scans.
Baselines
The naive reference is the frozen source-only model on raw target images. On the prostate benchmark, TASD reports this at 74.8% average Dice across five unseen sites (TASD). We retrain the same backbone on NCI-ISBI13 and report our own reproduced number beside theirs.
The strongest simple competitor is per-scan intensity histogram matching (plus z-scoring) to a source template. It is the obvious zero-training harmonizer and is exactly our method restricted to the low-pass filter. It uses the same masks and templates as our method. No reported result for it was found on Lacuna for these benchmarks, so we run it ourselves. Beating it means a statistically significant Dice gain (paired over scans) on the pooled target sites, and a larger gain on the sites we pre-classify as spectrally shifted.
The key ablation is the same pipeline with a fixed wavelet or steerable pyramid in place of the learned $\Omega$. It isolates whether learning matters. It uses identical thresholding and gain rules and the same tuning budget, a small grid over the clip range and threshold multiplier chosen on a source validation split only.
The strongest existing method is TASD, at 81.1% average Dice on prostate and 85.8% on fundus (TASD). The previous best single-domain generalization method, M-ADA, reached 78.2%. TASD updates batch-normalization parameters, so it is not admissible under a locked-model constraint. We treat it as an upper reference rather than a peer.
Success means closing at least half of the gap between our reproduced source-only number and TASD, while changing no weights and taking no gradient steps. Surpassing TASD would be a bonus rather than the bar. We also report the combination "ours then TASD" to see whether the two are complementary.
For the locked classifier setting the reference is the frozen DenseNet-121 pneumothorax model across NIH, Stanford and Beth Israel. The reported cross-site loss there is up to 0.18 AUC, and site fine-tuning recovers up to 0.23 AUC at a forgetting cost of 0.176 AUC (Regulating AI Adaptation). Beating the fine-tuning trade-off means recovering a meaningful fraction of the target loss with zero source-site degradation.
Why not just…?
Why not just histogram-match intensities? This is the real threat. For T2-weighted prostate MRI, much of the site shift may be global contrast, and histogram matching could capture most of the recoverable gain. We do not argue around this. Our method contains histogram matching as its low-pass channel.
The experiment settles it by stratifying target sites by measured spectral difference from the source (the ratio of high-band energy in the filter coefficients). The claim survives only if the margin over histogram matching grows with that spectral difference. The best places to test this are AMOS cross-scanner MRI and CT and the polyp sets. There, blur, glare and kernel differences dominate, as noted in Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation.
Why not just train a small convolutional input adapter at test time, pushing entropy or perturbation-consistency losses through the frozen model? It is strictly more expressive. INPUTADAPTER shows adapters in front of frozen predictors can work well (Adapt the Data, Not the Model), although with target labels. Without labels, an unconstrained image-to-image adapter can learn to produce confident but anatomically wrong inputs. It also needs backward passes through a 3D model for every volume.
We run this adapter with the same unsupervised loss and a matched compute budget. We report Dice, per-volume latency, and a hallucination check: Dice on held-out source scans passed through the adapter. If it clearly beats our method, the closed-form restriction is too tight. We would then test whether initializing a gradient-tuned $g$ from our closed-form solution gets the best of both.
Why not just update batch-norm statistics, as TASD and similar methods do? Doing so changes model parameters, which the locked-device setting rules out. It also needs gradient steps per image: TASD reports 56 ms per 2D slice, which multiplies across hundreds of slices in a volume. TASD itself shows that without a structural anchor such updates produce implausible shapes. We report it as the reference described above, not as an admissible competitor.
Why not just fine-tune? The cited regulatory analysis documents both why this is blocked in practice and its measured forgetting cost of 0.176 AUC on the original site (Regulating AI Adaptation). Our source-site preservation check reports the corresponding number for our front end.
Proposed Experiments and Decisive Checks
We test on four benchmarks.
- Prostate MRI. Train on NCI-ISBI13 and test on the five I2CVB and PROMISE12 sites, which gives a direct comparison with TASD's numbers.
- Fundus. Train on REFUGE and test on Drishti-GS, RIM-ONE-r3 and the third site. TASD reports 85.8%, but no reported baseline numbers were found on Lacuna for this benchmark, so we reproduce source-only and histogram matching ourselves.
- AMOS. Train on a subset of MRI scanners and test on held-out scanners, where the reported gap exceeds 20% mDice. This extends the method to 3D and is where latency matters most.
- Polyps. Train on Kvasir-SEG and test on ETIS and PolypGen. The cascaded DPN68×2 reports 76.19% Dice on ETIS. That is a trained architecture, not a locked-model method, so we use it only as context.
The pneumothorax setup tests whether the front end also helps a locked classifier.
Metrics are Dice or mDice (AUC for classification), per-volume adaptation latency, and change in source-site performance after harmonization. We also report gain range and variance per site, as evidence that the adjustment is auditable.
Three checks decide the outcome.
- Stratified comparison. The margin over histogram matching must increase with measured spectral shift. A flat or negative trend falsifies the core claim.
- Learned versus fixed filters. Learned $\Omega$ must beat the fixed wavelet ablation. Otherwise the sparse-coding contribution reduces to classical subband matching, and we would report it as such.
- Source preservation. Harmonizing held-out source scans must leave frozen-model performance unchanged within noise, directly contrasting with the forgetting cost of fine-tuning.
Secondary analyses compare threshold-only, gain-only, and both. They also compare global against tiled statistics on AMOS MRI, where bias fields are strongest.
References
- Regulating AI Adaptation: An Analysis of AI Medical Device Updates
- Learning Deep Analysis Dictionaries for Image Super-Resolution
- Projective dictionary pair learning for pattern classification
- PADDLE: Proximal Algorithm for Dual Dictionaries LEarning
- The Cosparse Analysis Model and Algorithms
- Separable cosparse Analysis Operator learning
- Analysis-Synthesis Dictionary Learning for Universality-Particularity Representation Based Classification
- Convolutional Dictionary Pair Learning Network for Image Representation Learning
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Single-Domain Generalization in Medical Image Segmentation via Test-Time Adaptation from Shape Dictionary
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation