Which Scale Sees the Harm? Kernel Bandwidth and Label-Free Monitoring of Frozen Medical Segmenters

A hospital that deploys a validated segmentation model usually cannot retrain it, and at a new scanner it has no labels with which to check it. What it can do is compare the model's internal features on incoming scans against features from the scans the model was validated on, and raise an alarm when they differ. The standard tool for that comparison is the maximum mean discrepancy (MMD), a distance between two sets of samples computed through a similarity function called a kernel. The most common kernel is a Gaussian bump whose width, the bandwidth, sets the scale at which differences count. This note argues that the bandwidth decides more than test power. It decides which scanner shifts the monitor reports. At one scale the monitor fires on harmless contrast changes. At another it misses a shift that costs 20 points of Dice, the standard overlap score for segmentations. We propose to measure this effect directly on multi-scanner abdominal and cardiac MRI. We then build a monitor whose mixture over many kernels is tuned, using only simulated corruptions of source data, so that its alarm level tracks expected performance loss rather than any statistical difference at all.

The claim has two parts. First, the default bandwidth rule (the "median heuristic," which sets the width to the median pairwise distance) produces metric inversion on real scanner shift: it ranks a harmful scanner as closer to the source than a benign one. This failure was documented for generative-model evaluation in Towards Robust Evaluation of Protein Generative Models, but nobody has looked for it in medical deployment. Second, a sequential test that bets across a portfolio of kernels, with portfolio priors calibrated on simulated scanner artifacts with known Dice cost, keeps its false-alarm guarantee. It also orders real unseen scanners by their true performance drop better than any single kernel, any unweighted kernel aggregate, or confidence-based monitoring.

What exists and what is missing

Two lines of work on the kernel side supply the method. MMD Aggregated Two-Sample Test combines tests over thousands of bandwidths with exact error control on fixed samples. Betting Across Kernels turns a kernel portfolio into an anytime-valid sequential test, meaning one that can be checked after every new sample without inflating false alarms. It tracks the best kernel online, paying only a $\log K$ cost for $K$ kernels. Both answer "is there any difference?" on synthetic data, corrupted MNIST and LLM outputs. Neither asks whether the difference they detect matters for a downstream model, and neither has been run on clinical images. We reuse the betting machinery unchanged and change what the portfolio is steered toward.

Two papers motivate the scale problem. The protein metrics study shows rankings flip below a bandwidth "sweet spot" and that stable estimates need on the order of 1,024 samples. Adaptive Kernel Selection for SVGD shows that the median heuristic breaks in high dimensions and that per-coordinate (anisotropic) bandwidths fix it. That second result is in sampling, not monitoring, and we borrow its anisotropic kernels as portfolio members.

The medical side supplies the regime. AMOS reports a drop of "over 20%" on MRI from unseen scanners. Adapt the Data, Not the Model formalizes the frozen-model constraint. It also finds that successful adaptation moved target data further from the source in Wasserstein distance. That finding constrains our design: raw distributional distance is not harm, so a monitor must be calibrated to harm explicitly. Improved post-hoc probability calibration for out-of-domain MRI segmentation shows that model confidence drifts under shift. Confidence is therefore an unreliable harm signal on its own, though it is the obvious competitor.

Prior kernel domain-adaptation work trains features to minimize MMD at a fixed, hand-tuned bandwidth and names bandwidth choice as unsolved. This includes Domain Adaptation on the Statistical Manifold, Rethink Maximum Mean Discrepancy for Domain Adaptation and Kernel Manifold Alignment. We deliberately do not propose another multi-bandwidth alignment loss. Losses that average over several Gaussian widths are already common practice in deep adaptation, and alignment requires changing the model, which the deployment setting forbids.

Research Program

The first step is a measurement study. We take a 3D U-Net trained on AMOS CT plus the MRI scanners from one subset. We extract features at three depths (early encoder, bottleneck, decoder) and pool them per volume and per organ-region patch. For every held-out scanner, and for ACDC (bSSFP) versus MS-CMRSeg (LGE) cardiac MRI, we compute MMD across a log-spaced grid of isotropic bandwidths and a set of anisotropic per-channel kernels. This gives a "shift spectrum": discrepancy as a function of scale and layer, for each site.

We pair each spectrum with the site's true Dice drop, measured with the labels these benchmarks provide. The core quantity is the inversion rate: the fraction of site pairs whose MMD ordering at a given kernel disagrees with their Dice-drop ordering. If the inversion rate at the median heuristic is near zero, the main hypothesis fails and we report that.

The second step is harm calibration without target labels. On source scans we apply TorchIO-style artifacts at graded strengths: bias fields, noise, blur, ghosting, motion, contrast remapping, and resolution changes. Because these are source scans, we know the Dice cost of each corrupted set exactly. We fit nonnegative weights $w_k$ over the $K$ portfolio kernels so that the weighted discrepancy $\sum_k w_k \,\mathrm{MMD}_k^2$ rank-correlates with simulated Dice cost.

These weights become the prior of the betting mixture. The anytime-valid guarantee holds for any prior fixed before monitoring, so calibration costs no validity. The open question is whether the weights transfer from simulated artifacts to real scanners. Testing that transfer is the decisive experiment.

The third step is deployment simulation. Target scans arrive one volume at a time. We record the number of volumes needed to raise an alarm, the false-alarm rate on in-distribution streams, and the alarm rate on shifts that are real but harmless. Harmless shifts are both constructed (simulated corruptions with Dice cost under one point) and real, namely sites whose measured drop is small. Volumes are the exchangeable unit for validity. Patches inside a volume are used only for power, with whole-volume permutation, so the roughly 1,024-sample stability concern applies at the patch level only.

Baselines

The do-nothing reference is no monitoring. The model is assumed fine at every site, so detection delay is infinite and the cost is the full unseen-scanner drop that AMOS reports.

The naive kernel reference is a median-heuristic Gaussian MMD permutation test at each layer. This is what most practitioners would run, and the measurement study's inversion rate is reported against it.

The strongest existing method is betting across kernels with a uniform prior over the same portfolio, as in Betting Across Kernels. On its own benchmarks it reports stopping at 0.65x the sample size of a hindsight oracle kernel. There is no reported result on Lacuna for kernel-portfolio monitoring on AMOS or cardiac MRI.

MMDAgg on fixed windows of 16 and 32 volumes is the strongest non-sequential competitor. It is run on the identical portfolio and the same windows.

All kernel methods share the same features, the same portfolio and the same false-alarm budget of 5%. The only thing the proposal adds is the harm-calibrated prior, so the comparison isolates that ingredient. Two ablations do the same for the other choices: isotropic-only versus anisotropic portfolios, and single-layer versus multi-layer features.

Beating the uniform-prior betting test means three things at matched 5% false alarms. The first is a higher Spearman correlation between alarm statistic and true Dice drop across held-out scanners and sequences. The second is a lower alarm rate on harmless shifts. The third is no worse detection delay on harmful ones.

Why not just…?

Why not just use the median heuristic? It is free and needs no portfolio. But the protein metrics study shows ranking flips below the right scale, and the SVGD paper shows the heuristic degrades with dimension. Our measurement study tests this directly. If median-heuristic inversion rates on AMOS are near zero, the proposal's premise is wrong and we will say so.

Why not just monitor the model's own confidence or prediction entropy? This is the most serious alternative. It needs no kernel at all, and it reads the model's output, which is closer to harm. The calibration paper shows that confidence becomes unreliable under exactly these MRI shifts, which suggests entropy may stay flat while Dice falls. That is a suggestion, not proof. We run entropy-shift and max-probability monitoring under the same sequential framework and false-alarm budget. If entropy matches the calibrated kernel monitor in harm correlation, the contribution narrows to the inversion finding, and we will report it that way.

Why not just train a classifier to tell source from target, or use sliced Wasserstein distance? Both avoid bandwidths. But both answer "is there a difference?", not "does it hurt?" The input-adaptation paper's finding that helpful adaptation increased Wasserstein distance is direct evidence that untuned distances need not track harm. We include both as monitors, calibrated with the same harmless-shift protocol, so the experiment shows whether harm calibration rather than the kernel family is what matters. If a harm-calibrated classifier test does as well, that is the honest conclusion.

Why not just apply targeted augmentation so shift stops mattering? Out-of-Distribution Robustness via Targeted Augmentations lifted Camelyon17-WILDS OOD accuracy from 75.3% to 90.5%. But augmentation changes the model, which the deployment setting forbids, and AMOS still shows a drop of over 20% on unseen MRI scanners. Monitoring complements augmentation. Augmented models are monitored too, and we report whether augmentation shrinks the set of harmful shifts.

Decisive Checks

Three results settle the proposal. The first is whether median-heuristic MMD inverts the AMOS scanner and bSSFP→LGE orderings relative to true Dice drop at a non-trivial rate. The second is whether portfolio weights fit only on simulated source artifacts raise held-out real-site harm correlation over the uniform-prior betting test, under leave-one-scanner-out evaluation, while keeping the 5% false-alarm rate on in-distribution streams. The third is whether that gain survives against entropy monitoring and a harm-calibrated classifier test.

As a check on generality beyond MRI, we repeat the transfer test on Camelyon17-WILDS hospitals, using stain-jitter simulations for calibration and per-hospital accuracy as the harm measure. A positive answer on all three checks gives clinics a monitor that alarms on what hurts. A negative answer on the second check tells the field that simulated artifacts do not span real scanner shift. That is also worth knowing before anyone deploys a simulation-calibrated monitor.