Anatomy-Shaped Contrast Normalization for Locked Medical Segmenters

When a medical segmentation network trained at one hospital is run at another, much of what goes wrong is an intensity problem: a different scanner or camera brightens some regions, flattens others, and changes the noise level. The network itself is often legally frozen, because changing a certified model's weights means certifying it again. One safe remedy is to fix the image rather than the model, by re-normalizing local contrast before the frozen network sees it. Current local normalizers compute brightness statistics over rigid square windows or round blurs. Near thin, winding structures such as retinal vessels, those windows straddle vessel and background, so the statistics they produce mix two tissues. We propose a learned input adapter that computes local mean and spread over a window that bends to follow the anatomy, in the way a deformable convolution (a convolution whose sampling points are shifted by small learned offsets) bends its kernel. The adapter is then restricted to a per-pixel affine remapping of each pixel's own value.

That restriction is the point. The adapter cannot paint in vessels or lesions. It can only rescale each pixel against statistics gathered from a neighborhood shaped like the structure that pixel belongs to. We claim three things. First, this anatomy-shaped normalization recovers more of a frozen segmenter's cross-site accuracy than fixed CLAHE (contrast-limited adaptive histogram equalization, the standard tile-based local contrast method) or learned isotropic normalization. Second, the gain concentrates on thin and low-contrast structures, where rigid windows mix tissues. Third, clinicians judge the adapted images to be faithful, and unrestricted generative adapters do not reliably pass that test. Human perceptual preference therefore enters this proposal as an acceptance constraint, not as a training loss.

Core Claim

Picture a pixel sitting on a two-pixel-wide vessel. A $15\times15$ rigid window around it is mostly background, so its local mean is a background mean, and the vessel's normalized contrast depends on how bright the background happens to be at that site. If the window's sample points slide along the vessel, the statistics describe the vessel's own neighborhood and its immediate flanks. The normalized value then becomes roughly invariant to the site-specific gain and offset that scanners apply. We write the adapter output at pixel $p$ as

$$\mu(p)=\sum_{k=1}^{K} a_k(p)\, x\big(p+q_k+\Delta q_k(p)\big),\qquad y(p)=\gamma(p)\,\frac{x(p)-\mu(p)}{\sqrt{\sigma^2(p)+\epsilon}}+\big(1-\gamma(p)\big)\,x(p).$$

Here $x$ is the input image and $q_k$ are $K$ base sample positions spread over a fixed footprint. $\Delta q_k(p)$ are learned fractional offsets, read off by bilinear interpolation as in deformable convolution, and $a_k(p)$ are softmax weights. $\sigma^2(p)$ is the correspondingly weighted local variance, $\epsilon$ is a noise floor that plays the role of CLAHE's clip limit, and $\gamma(p)\in[0,1]$ is a learned blend toward the raw image. A small network predicts $\Delta q$, $a$ and $\gamma$ from a rigidly pre-normalized copy of the image. That ordering avoids having offsets computed on the very intensities they are meant to correct. Because $y(p)$ is affine in $x(p)$ with smoothly varying coefficients, new edges can only appear where the coefficients change abruptly. We penalize offset and coefficient roughness to keep that from happening.

Relation to Prior Work

Three papers supply the mechanism, and each stops short of this design. ALCN learns to mix isotropic Gaussian filters of different scales per patch, trained jointly with a detector. Under extreme lighting it raised detection AUC from 0.611 for fixed Difference-of-Gaussians to 0.787. Its kernels are round, it is tested on natural images, and the detector is trained with it rather than frozen. Local Context Normalization normalizes features over rigid $p\times q$ windows using summed-area tables. It beat Group Normalization by more than 6% mIoU on aerial imagery and reached 77.49% mIoU on Cityscapes against 76.22% for batch normalization. Its windows cannot bend, and the constant-time integral trick it relies on does not survive deformation, so we pay a $K$-sample cost per pixel instead. DUNet is the nearest point of contact. It feeds fixed CLAHE-preprocessed fundus images into a U-Net with deformable convolutions, reaching AUC 0.9856 on DRIVE and 0.9868 on STARE. Its contrast step is fixed and uncoupled from its offsets, and it was not evaluated across sites. We move the deformation from the segmenter's features into the normalization statistics, and we train it in front of a frozen model.

Two papers motivate the locked-model framing. Adapt the Data, Not the Model shows that adapting inputs to frozen clinical predictors can beat fine-tuning, with a +14.1 AUCPR gain on eICU kidney injury prediction. It works on time series and uses a free-form encoder-decoder. In images, such a decoder could hallucinate anatomy, and our restricted transformation family is designed to rule that out. CAR shows that random-convolution contrast augmentation needs an explicit invariance regularizer to work. That result constrains our training recipe: we simulate site shifts with random local intensity transforms, and we judge the adapter by whether the frozen network's output on shifted images matches its output on clean ones.

Two further papers constrain our expectations. DCAC adapts kernel weights to domain and content but keeps a rigid sampling grid, and it retrains the whole segmenter. Its numbers therefore act as an upper reference that an unlocked model can reach. AMOS reports that performance drops by more than 20% on unseen MRI scanners even with multi-scanner training. Intensity mismatch is large, but some of that drop is surely not intensity, and no input normalizer will recover that part.

Finally, When does perceptual alignment benefit vision representations? shows that tuning backbones to human similarity judgments improves dense prediction and transfers to diabetic retinopathy. We have no medical human-similarity data to train on, so we do not borrow its loss. We take its premise that human perception tracks structural plausibility, and we use clinician judgments to test whether the adapted images remain trustworthy.

Research Program

We first build the adapter as a drop-in 2D module. It uses $K=25$ sample points over a $9\times9$ base footprint and an offset predictor of about 50k parameters. Training runs on source-site images only, with the downstream segmenter frozen. Each source image is corrupted by a random smooth bias field, a random gamma curve, local contrast compression and site-like noise. The adapter is trained so that the frozen segmenter's prediction on the corrupted image matches the label, plus a consistency term toward the segmenter's prediction on the clean image. Two penalties are added: one on offset and coefficient roughness, and one that keeps the effective window from collapsing below a minimum spread. Without the second penalty, $\sigma(p)$ goes to zero and noise is amplified, which is the same failure CLAHE's clip limit exists to prevent.

Second, we analyze where the gain comes from. We bin pixels by vessel width and distance to boundary. We check whether the learned offsets follow vessel centerlines, as DUNet's feature offsets do. We also measure offset stability across sites by comparing the offset fields on the same anatomy rendered under different simulated scanners.

Third, we run a reader study. Two ophthalmologists and one radiologist see raw, fixed-CLAHE, isotropic-adapter, generative-adapter and deformable-adapter versions of the same target-site images. They make blinded pairwise preference choices and give a binary fidelity judgment ("I would read from this image").

Baselines

The naive reference is the frozen segmenter applied to raw target-site images. Every adapter must beat it, and its gap to the unlocked upper reference tells us how much there is to recover.

The practical default is fixed CLAHE as preprocessing, following DUNet's pipeline of grayscale conversion, CLAHE and gamma correction. We tune clip limit and tile size by grid search on held-out source data with the same compute budget as the adapter's tuning. Beating it requires a gain in mean cross-site Dice that holds over three seeds and on at least three of four target datasets.

The most important ablation keeps everything in the deformable adapter but fixes $\Delta q=0$. We test this in two versions, rigid windows in the style of LCN and multi-scale isotropic Gaussians in the style of ALCN, with parameter counts and FLOPs matched by widening the predictor. If the deformable adapter does not beat both, the anatomy-shaping claim fails.

The strongest free-form competitor is an INPUTADAPTER-style encoder-decoder trained with the identical source-only recipe. It may match or beat us on Dice. We count our method as better only if it reaches comparable Dice, within one Dice point, with higher clinician fidelity and fewer spurious structures in vessel-free regions.

The strongest existing domain-generalization methods serve as references with unlocked weights. On fundus optic cup and disc, DCAC reports 88.47% average Dice. On cross-site retinal vessels, ErrorNet reports a cross-domain average Dice of 71.0. We will rerun DCAC on our splits. Coming within two Dice points of DCAC while keeping the segmenter frozen would count as success. Exceeding ErrorNet's 71.0 under its protocol would be a clear win.

Why not just…?

Why not just tune CLAHE per site? This is a real threat for fundus images, whose shifts are dominated by illumination. Per-site tuning needs labeled target data, which is rarely available at deployment, and a single global setting cannot avoid tile windows straddling vessel and background. ALCN's evidence that fixed filtering lags adaptive filtering comes from natural images, not medicine. The tuned-CLAHE baseline, including an oracle version tuned on target labels, settles the question directly. If the oracle matches our adapter, we will report that the deformation is not worth its cost.

Why not just train the segmenter with heavy intensity augmentation or random convolutions? In the locked regime the segmenter is already fixed, but a vendor could have trained it this way in the first place. CAR shows that random convolution alone underperforms without an invariance term, and AMOS shows that large cross-scanner drops survive multi-scanner training. We will freeze two segmenters, one trained plainly and one with heavy augmentation, and report the adapter's gain on each. If the gain vanishes on the augmented model, the adapter's value is limited to legacy models, and we will say so.

Why not just put deformable convolutions inside the segmenter, as DUNet does? That requires retraining, which the locked regime forbids. Even setting regulation aside, it leaves open whether deformation helps more in the statistics or in the features. We include an unlocked DUNet-style model with CLAHE as a reference to show what feature-level deformation buys when retraining is allowed.

Why not just use a generative image-to-image adapter? It is more expressive and may score higher Dice. It can also invent vessels. Our hallucination checks give the answer here: spurious segmented structure in vessel-free regions, and clinician fidelity judgments.

Proposed Experiments

The primary setting is retinal vessels. We train on DRIVE, freeze a U-Net and a DUNet-style segmenter, and test on STARE, CHASE_DB1, HRF and ARIA. We report Dice and AUC, plus a connectivity error defined as the difference in the number of connected vessel components from the ground truth. The secondary setting is optic cup and disc across the four fundus sites in DCAC, with leave-one-site-out evaluation and Dice and average surface distance. The third setting is the six-site prostate MRI benchmark, using 2D slices. Prostate structures are blob-like, so we predict a smaller deformable advantage there. Finding the same gain on prostate as on vessels would suggest the benefit is not anatomy-shaping but extra capacity, and that check is deliberate.

The decisive results come in three parts. First, the deformable adapter must beat the matched rigid and isotropic ablations on vessels. Second, the stratified analysis must show the advantage growing as vessel width shrinks. A flat profile across widths would falsify the mechanism even if mean Dice improved. Third, clinicians must prefer, or at least accept as faithful, the deformable adapter's images at a rate no lower than fixed CLAHE, while the generative adapter falls below that rate. We will also apply the adapter to clean source images. A transform that genuinely normalizes should leave those near identity, within half a Dice point. Finally, we will measure runtime per image, since the $K$-sample cost replaces LCN's constant-time windows.