Medical image segmentation models that work well at the hospital where they were trained often fail at a new hospital with different scanners. On the AMOS abdominal benchmark, Dice overlap drops by more than 20% on MRI from unseen scanners. Regulation makes the fix harder. Once a model is cleared as a medical device, its weights are locked, so the usual remedy of fine-tuning on target-site data is unavailable. One way to respect the lock is to leave the model frozen and learn a small correction applied to incoming images. We propose making that correction a linear image filter combined with an intensity remapping, and training it as a product of several linear factors that are multiplied back into a single filter at deployment. The deployed adapter is therefore one auditable convolution kernel plus one intensity curve. The training-time depth is a deliberate choice of implicit bias, meaning the preference that gradient descent shows among the many solutions that fit the few labels a target site can afford to annotate.
The idea works because of a structural coincidence. The cleanest theorems about how depth changes learning hold for diagonal linear networks. In these models each output coordinate is a product of per-layer scalars, and depth biases gradient descent toward sparse solutions at realistic initialization scales. A stack of convolutions with no nonlinearity between them is exactly such a model in the Fourier basis: each frequency is multiplied by a product of per-layer coefficients. Our core claim is that when an input filter is trained with a depth-$L$ factorization, the theory predicts it will find sparse-in-frequency corrections, which change only the few frequency bands where the new scanner differs, and that this preference reduces the number of target-site annotations needed to recover segmentation accuracy behind a frozen predictor. The proposal tests whether depth does anything that a well-tuned explicit sparsity penalty does not, and treats a negative answer as a real possible outcome.
Core Claim and Mechanism
Let $x$ be an input image (2D or 3D) and $f$ the locked segmenter. The adapter outputs $\tilde x = k * \phi(x)$. Here $\phi$ is a per-voxel intensity remapping built from a fixed spline basis with learnable coefficients, $*$ is convolution, and $k$ is a learnable kernel. During training only, we write the kernel's frequency response as a product of $L$ factors:
$$\hat k(\omega) = \prod_{l=1}^{L} \hat k_l(\omega),$$
where $\omega$ indexes spatial frequency and $\hat k_l$ is the Fourier transform of the $l$-th factor kernel. The spline coefficients of $\phi$ are factored the same way, as elementwise products of $L$ vectors. After training, the factors are multiplied out, and the deployed object is a single kernel and a single curve, regardless of $L$.
Two theoretical results supply the mechanism. Kernel and Rich Regimes in Overparametrized Models shows that for depth-$D$ diagonal models, the initialization scale $\alpha$ selects what gradient flow implicitly minimizes. It moves from an $\ell_2$ norm in the "lazy" regime (large $\alpha$, weights barely move) to an $\ell_1$-like norm in the "rich" regime (small $\alpha$). Depth relaxes the initialization needed to reach the sparse end from exponentially small ($\sim e^{-D}$) to polynomially small ($\sim D^{-c}$). On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization shows that factoring a linear map into a product of matrices is equivalent to training the single map with a data-dependent preconditioner. That preconditioner cannot be reproduced by any fixed added regularizer, and its effect depends on depth and not on width.
Most uses of these theorems in practical networks are loose analogies. Here the analogy is unusually tight, because the trainable object really is linear and really is diagonal in a known basis. Only the loss is nonstandard: it is Dice measured through a frozen nonlinear network. That gap is a real caveat. The file notes that extending these results beyond squared loss is open, so whether the predicted bias survives is an empirical question we measure directly.
Relation to Prior Work
The proposal reuses its deployment pattern from Adapt the Data, Not the Model. That paper trains an input-space adapter in front of a frozen ICU time-series predictor, matches or beats end-to-end fine-tuning on eICU (+14.1 AUCPR on acute kidney injury), and finds that adapters partly transfer across frozen predictors. Two things about it motivate us. Its adapter is a nonlinear network that needs labeled target data. It also works on time series, not images. We change the adapter's form, making it linear, collapsible and inspectable, and we ask whether the training-time factorization reduces how many target labels are needed.
On the imaging side, AMOS (AMOS) motivates the setting by documenting the MRI scanner gap and the fact that small organs degrade most. The robustness architectures in the file all change the predictor itself, which a locked device forbids:
- cascaded encoder-decoders (Double Encoder-Decoder Networks);
- hybrid transformer decoders (TransNetR);
- deformable 3D backbones (DeformUX-Net).
We use them only as frozen predictors.
Simulated Overparameterization is the closest precedent for "extra parameters at training time, none at inference." It does this with stochastic majority kernels in classifiers, not with depth, and not for adaptation. How Muon's Spectral Design Benefits Generalization shows that in deep linear networks, depth narrows the gap in learning speed between dominant and minority components. That suggests a secondary prediction: weak, low-energy frequency bands that differ between scanners should not be learned last.
Research Program
We first train locked predictors on source scanners using standard augmentation. The predictors are a 3D UNet and DeformUX-Net, the latter via MASILab/deform-uxnet. These weights are then frozen for the rest of the study.
For each target site we sample $K \in \{1, 2, 5, 10\}$ labeled target volumes or images and fit adapters across a grid:
- depth $L \in \{1, 2, 3, 4, 6\}$;
- initialization scale $\alpha$ swept over several orders of magnitude;
- optimizer, either plain gradient descent or Adam.
Each fitted adapter is evaluated on the remaining target data. We also look inside the adapter. For each run we measure what fraction of the correction's spectral energy sits in its top 5% of frequencies, how that fraction moves with $L$ and $\alpha$, and whether the $\alpha$ needed to reach a given sparsity scales polynomially or exponentially with $L$, as the theory predicts.
We also test portability. An adapter fitted behind the UNet is transplanted unchanged behind DeformUX-Net, mirroring the cross-predictor transfer reported for time series. The idea is that a correction which fixes the scanner, rather than one that exploits a particular network, should transfer.
Baselines
The do-nothing reference is the locked predictor applied to target data with no adapter. Every gain is measured relative to it. The natural no-learning step beyond that is classical intensity standardization: histogram matching of target images to the source intensity distribution, with no labels. Beating it shows the labels are being used for something.
The strongest competitor is an image analogue of INPUTADAPTER (Adapt the Data, Not the Model): a small nonlinear convolutional network placed before the frozen segmenter and trained on the same $K$ target labels with the same Dice-plus-cross-entropy loss. It is the strongest because it is the only existing method in the file that respects the locked-predictor constraint and has been shown to match fine-tuning. Fairness rules:
- It receives the same target samples and the same number of gradient steps.
- It receives the same hyperparameter search budget, counted in configurations, as the factored adapter's combined search over $L$ and $\alpha$.
- Its capacity is swept from tiny to moderate.
There is no reported result found on Lacuna for input-space adaptation of frozen image segmenters. The only reported number for this family is +14.1 AUCPR on acute kidney injury with eICU as target. On the imaging benchmarks, the reference points are therefore the ones the file reports:
- 76.19% Dice on the unseen ETIS polyp set for the Double Encoder-Decoder without adaptation (Double Encoder-Decoder Networks);
- 89.04% CT mDice for 3D UNet on AMOS (AMOS);
- a drop of over 20% on unseen MRI scanners on AMOS.
Our own frozen-model numbers will be reported alongside these, not equated with them.
Two ablations isolate the key ingredient. The first is the same deployed adapter family, one kernel and one intensity curve, trained directly with $L=1$ and a tuned optimizer. The second is $L=1$ with an explicit $\ell_1$ penalty on the kernel's frequency response and on the spline coefficients, with the penalty strength tuned on held-out target data under the same budget. This second ablation directly tests whether depth only reproduces a sparsity regularizer.
Full fine-tuning of the segmenter on the $K$ target samples is reported as a non-compliant upper reference.
The factored adapter "beats" a baseline only under three conditions. It must exceed the baseline's target Dice at $K \le 5$ by more than the across-seed standard deviation. It must do so on both the MRI and the polyp benchmarks. The gain must be largest on small structures: adrenal glands and pancreas on AMOS, and small polyps.
Why not just…?
Why not just augment harder when training the original model? Training-time augmentation can only anticipate shifts that someone imagined in advance, and changing it means retraining a locked model. Our locked predictors are already trained with standard augmentation, so whatever augmentation buys is included in the do-nothing reference. The AMOS MRI drop of over 20% is reported for conventionally trained models. The open question is what can be recovered after lock, which augmentation does not address.
Why not just use a nonlinear CNN adapter like INPUTADAPTER? This is a real threat, and it is the main baseline. With enough target labels it may well win, since it can express corrections a linear filter cannot. Our bet is narrower. At $K \le 5$ a flexible adapter can overfit, or learn image edits that fool the frozen network rather than undo the scanner difference. A collapsible linear correction is also something a reviewer or regulator can inspect as a frequency response. If the nonlinear adapter wins at every $K$, including $K=1$, and also transfers better across predictors, we will report that the linear family is too restrictive.
Why not just add an explicit sparsity penalty to a single-filter adapter? This is the sharpest objection, and the proposal is designed to answer it. On the Optimization of Deep Networks proves that the dynamics of depth cannot be reproduced by a fixed regularizer. That theorem is about the path of optimization, not about which final solution generalizes better. If the tuned $\ell_1$ baseline matches the deep factorization in accuracy and sparsity, the honest conclusion is that "sparse frequency correction" is the useful idea and depth is just an equivalent way to tune it. Depth would then keep two smaller advantages: fewer steps to converge, and no penalty strength to tune per site, which matters when a site has five labels and no validation set. We will report both outcomes either way.
Why not just fine-tune the segmenter? Under the device regulations described in Adapt the Data, Not the Model, changing the weights invalidates clearance. We include fine-tuning only to show how much accuracy compliance costs.
Proposed Experiments and Decisive Checks
AMOS MRI. Train on a subset of the scanners and treat each held-out MRI scanner as a target site. We report:
- mean Dice and per-organ Dice, especially adrenal glands, pancreas and duodenum;
- 95th-percentile Hausdorff distance;
- label-efficiency curves over $K$;
- the number of steps needed to reach 90% of final Dice.
Polyp segmentation. Train on Kvasir-SEG and CVC-ClinicDB, then adapt to ETIS-LaribPolypDB, CVC-ColonDB and each PolypGen center. This gives a cheap 2D setting with many sites, across five seeds per configuration.
Three checks decide the outcome.
- Accuracy. Depth $L \ge 3$ at standard initialization must beat $L=1$ with a tuned $\ell_1$ penalty at $K \le 5$ on held-out sites. If it fails, the depth claim fails.
- Mechanism. The learned corrections must become sparser in frequency as $L$ grows or $\alpha$ shrinks, with the needed $\alpha$ scaling polynomially in $L$. If it does not, the theory does not explain what happened, even if accuracy improves.
- Portability. Adapters fitted behind one frozen predictor must keep most of their gain behind the other. This separates a scanner correction from a model-specific trick.
A positive result on all three would give a locked medical model a site-specific correction that can be fitted from a handful of annotated scans and audited as a single filter. A failure on the first check would still clarify whether depth-induced bias offers anything beyond explicit regularization in a setting where the theory applies almost exactly.
References
- Kernel and Rich Regimes in Overparametrized Models
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing
- DeformUX-Net: Exploring a 3D Foundation Backbone for Medical Image Segmentation with Depthwise Deformable Convolution
- Simulated Overparameterization
- How Muon’s Spectral Design Benefits Generalization: A Study on Imbalanced Data