A Deployment-Time Dial for Loss Weights in Medical Segmentation

Medical segmentation models are trained with composite losses, for example an overlap term (Dice), a pixel-wise cross-entropy term, and sometimes a penalty for anatomically impossible shapes or for features that reveal which scanner produced the image. The weights on these terms are chosen once, on the training hospitals' validation data, and then frozen. We think this is the wrong moment to choose them. The best trade-off between voxel overlap and anatomical plausibility on a clean 3T research scan is probably not the best trade-off on a motion-corrupted 0.55T low-field scan from another hospital. The proposal has three steps. Train a small number of "endpoint" models that branch from one shared checkpoint and differ only in their loss weights. Ship those weights to each deploying hospital. Let the hospital choose a point on the straight line, or small simplex, between them, using only its own unlabeled scans. No patient data leaves the site, no retraining happens there, and the deployed artifact is a single network at ordinary inference cost.

This works only if two empirical facts hold, and both are testable. The first is linear mode connectivity: models fine-tuned from a shared starting point can be averaged in weight space without a loss spike. It is well documented for task and hyperparameter variants, but not for loss-weight variants of a single 3D segmentation task. The second is that the best loss weighting actually differs across clinical sites. If it does not, a single tuned weight is enough and the idea should be abandoned. The proposal is designed so the second fact is measured first.

Core Claim

Our claim has two parts. Branching from a shared 3D segmentation checkpoint and fine-tuning under different loss weightings yields endpoints whose linear interpolations closely track models trained directly at intermediate weightings. Choosing the interpolation coefficient per target site, using label-free or few-label criteria, then gives better topology and boundary quality under scanner and site shift than any single loss weighting tuned on source data, without losing Dice.

Concretely, let $\theta_a$ and $\theta_b$ be endpoints fine-tuned from a common checkpoint $\theta_0$ with loss weights $\alpha_a$ and $\alpha_b$ on an auxiliary term. Here $\alpha$ is the coefficient on, for example, a topology penalty in $\mathcal{L} = \mathcal{L}_{\text{Dice+CE}} + \alpha\,\mathcal{L}_{\text{aux}}$. The interpolated model is

$$\theta(\lambda) = (1-\lambda)\,\theta_a + \lambda\,\theta_b, \qquad \lambda \in [0,1],$$

and a deploying site $s$ selects

$$\lambda_s = \arg\max_{\lambda} \; Q_s(\theta(\lambda)),$$

where $Q_s$ is a quality score computed on site $s$'s own scans. The main version of $Q_s$ is label-free: the agreement between the predicted segmentation's Euler characteristic (a count of connected pieces minus holes, which summarizes shape topology) and the value expected from training anatomy, combined with prediction entropy. A variant uses a handful of labeled site scans, the kind of quality-assurance set a hospital already curates.

This differs from the closest work in specific ways. Model soups average hyperparameter variants into one fixed model chosen on source validation data. We keep the variants as a dial and set it per site. Robust fine-tuning of zero-shot models (WiSE-FT) interpolates between a pretrained and a fine-tuned model. Our endpoints differ in the objective, not in the amount of adaptation. EXTRA fits a curved low-loss path between endpoints specialized to different tasks. We study endpoints of a single task under different loss weightings, and we choose the operating point at deployment rather than at training time. MedSAMix searches layer-wise merges between a generalist SAM and medical specialists with Bayesian optimization on labeled validation data. Its merges combine different training histories, not different loss weightings, and its choice is made once per task rather than per hospital.

Why This Is Plausible, and What Constrains It

Three results motivate the proposal. What is being transferred in transfer learning? shows that fine-tuning from a shared checkpoint stays inside one low-loss basin. On the Emergence of Cross-Task Linearity shows that, in that regime, weight interpolation approximately interpolates internal features. That is the mechanism by which a weight-space dial could behave like a loss-weight dial. Training-Free Model Merging for Multi-target Domain Adaptation shows that linearly merged, separately adapted driving-scene segmenters reach 56.3% mIoU against 56.2% for pooled-data training, provided they share a backbone and normalization statistics are merged carefully. This is direct evidence that segmentation models survive weight averaging across domains.

Two results identify the problem. The Sensitivity of Multi-Objective Loss Balancing documents how fixed loss weights fail through gradient-magnitude dominance and directional conflict. EXTRA reports that dynamic balancers such as Nash-MTL and FairGrad often fail to beat plain fixed weighting. Together these suggest the problem is not solved by smarter balancing during training, which favors moving the choice after training.

On the medical side, the FeTA 2024 challenge found two things. Image quality (about 0.10 Dice spread) and acquisition site (about 0.075) dominate degradation. Models with similar Dice can differ greatly in topological error measured by Euler characteristic difference. This is exactly a trade-off whose optimum could plausibly move with site and image quality. AMOS reports a drop of over 20% on MRI from unseen scanners.

Two findings constrain the design. Are we Merging the Right Models? shows that long fine-tuning can degrade mergeability, so endpoint fine-tuning length is a controlled variable, not a free one. The heterogeneous-merging literature, Training-free Heterogeneous Model Merging and Deep Neural Network Fusion via Graph Matching, shows that dropping the shared-ancestry assumption requires explicit alignment of neurons and layers. Even then, these methods stay within one architecture family and do not merge convolutional networks with Vision Transformers. We therefore do not attempt CNN-to-Transformer weight merging. We treat architectural heterogeneity as a boundary to be measured, as described below.

Research Program

The base model is a 3D U-Net in the nnU-Net configuration, trained on the pooled source sites with the standard Dice plus cross-entropy loss. We use this rather than a Vision Transformer because AMOS reports the 3D U-Net ahead of Swin-UNETR on CT (89.04% vs 86.32% mDice), and nnU-Net variants dominated FeTA. From this checkpoint we branch endpoints by continuing training for a fixed, short budget under different weights on one auxiliary term.

On FeTA the auxiliary term is a differentiable topology-aware penalty that discourages spurious holes and components. On AMOS it is an adversarial scanner/modality-invariance term of the kind used in Domain Separation Networks, whose conflict with the task loss is well documented. When both terms are used together, three endpoints span a two-dimensional simplex of interpolations. There, rather than grid-searching, we fit the quadratic surrogate of MAP to a few dozen evaluated mixtures. MAP reports surrogate $R^2$ above 0.95 near shared checkpoints, though only for classification and 2D chest X-ray. Batch- or instance-normalization statistics are handled with the buffer-merging procedure of the multi-target adaptation paper, so the comparison isolates the weights themselves.

The work has three analyses. The first measures whether the target exists: we train a dense grid of models at many fixed loss weights and record, per held-out site, which weight is best. This is expensive but is the ground truth everything else is judged against. The second measures how faithfully the dial tracks the grid. At each $\lambda$ we compare $\theta(\lambda)$ with the directly trained model at the matching interpolated weight, using voxel-level prediction agreement and the metric gap. This tests whether loss-weight space maps onto weight space approximately linearly. The third measures whether a site can find its own setting: we correlate the label-free score $Q_s$ and the few-label score with the oracle choice, per site.

A final boundary study connects to heterogeneous merging. Because nnU-Net configures depth and width per dataset, two sites running it independently obtain networks of the same family with different shapes and no shared ancestry. We ask whether the dial survives in that setting after alignment with Elastic Neuron Zipping and layer alignment from HeteroZ, or permutation matching from GAMF. We measure the loss barrier along the aligned path. We expect a substantial barrier. Reporting how much alignment recovers tells practitioners whether a shared checkpoint is mandatory, which is the practical constraint this idea imposes on multi-hospital deployment.

Baselines

The do-nothing reference is the standard practice. A single model is trained with the loss weight chosen on pooled source validation data, then deployed unchanged everywhere. A second naive reference is default nnU-Net with no auxiliary loss. These show whether any per-site choice helps at all.

The strongest single-model competitor on our question is the dense grid of separately trained models with per-site selection using the same selection signal. It is the obvious "just train more models and pick one" answer. We run it with the same selection budget (same unlabeled scans, same few labeled scans) and report both its per-site oracle and its realistic selection. The dial beats it if it matches the grid's selected performance within noise while training three or fewer endpoint runs instead of a dense grid. It also beats it if it exceeds the grid because the interpolated points are smoother than any individually trained one, as Model soups observed for hyperparameter averages.

The strongest published systems on these benchmarks set absolute reference points. They are not directly reproducible comparisons, because our site splits differ. On FeTA 2024, the 0.55T low-field test set result reported by the challenge paper is Dice 0.86, HD95 1.69, ED 6.26. The winning team's ED is 20.9, about 50% better than the next teams at comparable Dice. On AMOS, Extending nnU-Net is all you need reports Dice 90.13 on CT and 89.06 on CT+MRI. This comes from a 10 to 15 model ensemble needing 50 to 60 hours of inference on an RTX 3090, and the paper reports no single-model unseen-scanner figure. For the AMOS unseen-scanner MRI setting, only the "over 20% drop" statement exists; no reported numeric result was found on Lacuna for AMOS unseen-scanner MRI Dice. We will retrain the extended nnU-Net recipe as a single model on our splits so the comparison uses matched data and compute.

The strongest merging competitor is a MedSAMix-style search. We apply its layer-wise Bayesian-optimization merge search over our own endpoints, with the same few labeled site scans, and allow the same number of evaluations as our selector. MedSAMix reports a 6.67% Dice gain over MedicoSAM on its 25-task suite, but no reported result was found on Lacuna for MedSAMix on FeTA or AMOS. This tests whether a richer, layer-wise merge search beats a one-parameter dial. If it does, the extra degrees of freedom are worth their cost.

The preference-conditioned competitor follows Pareto Merging: a single model with a low-rank component conditioned on the loss-weight value, trained once. It reports 85.9% average accuracy on its 8-dataset CLIP benchmark. We give it a training budget equal to our endpoint runs combined.

The site-adaptation competitor is normalization-statistics recomputation on target scans, as in the buffer merging of the multi-target adaptation paper. We test it alone and combined with the dial.

Three ablations isolate the key ingredients. Shared-checkpoint branching versus independently initialized endpoints, with and without HeteroZ or GAMF alignment, isolates the role of a shared basin. Continuous interpolation versus choosing only among the endpoints themselves isolates the value of the continuum. Label-free versus few-label versus oracle selection isolates how much of the gain a hospital can actually realize.

Why not just…?

Why not just tune the loss weight once, carefully, on source data? This is the real threat. If the dense-grid analysis shows that the best weight is the same across FeTA's in-domain, out-of-domain high-field, and 0.55T sites, and across AMOS's seen and unseen scanners, then per-site selection has nothing to exploit. In that case we would report the null result and drop the per-site claim. FeTA's finding that quality and site, not pathology, drive the Dice spread and that topology varies independently of Dice is suggestive, but it does not establish that the optimum moves. That is why this measurement comes first.

Why not just train a model at each of several loss weights and let each hospital pick one? This works if the grid is fine enough. Our claim is that interpolating two or three branched endpoints reproduces a fine grid at a fraction of the training cost, and that a hospital can choose continuously. The tracking-gap analysis settles whether interpolation is a faithful substitute. If interpolated models fall off the directly trained curve, discrete selection wins and we will say so.

Why not just ensemble the endpoints, or ensemble across architectures? Ensembling avoids every weight-space assumption and can combine a U-Net with a Transformer, which our method cannot. Its cost is concrete: the AMOS-winning ensemble needed 50 to 60 hours of inference. We include the endpoint ensemble as an upper reference with its inference cost reported, so readers can see how much of the ensemble's gain a single interpolated network retains.

Why not just distill all site or endpoint models into one student, or train a weight-conditioned network? Multi-teacher distillation reintroduces a hard-label versus soft-label weighting, the same sensitivity documented in object-detection distillation by Chen et al.. It also requires a shared training set. The conditioned network, our Pareto Merging baseline, is a genuine competitor that needs one training run. The matched-budget comparison decides between it and the dial.

Proposed Experiments and Decisive Checks

On FeTA 2024 we train on the two training institutions (Zurich and Vienna). We evaluate on the in-domain, out-of-domain high-field, and 0.55T low-field sets where available. If test sets are unreleased, we hold out one institution and stratify by image quality. Metrics are Dice, 95th-percentile Hausdorff distance, and Euler characteristic difference, reported per site and per quality bin.

On AMOS we train on CT and the MRI scanners we designate as seen, then evaluate on held-out MRI scanners. Metrics are mean Dice over 15 organs, HD95, and the rate of left-right organ confusion, a failure mode the extended nnU-Net needed post-processing to fix.

Each method receives the same training compute, counted in total GPU-hours across all runs. Each also receives the same target-site selection budget: a fixed number of unlabeled scans and, in the few-label condition, five labeled scans.

Four checks are decisive. The project proceeds only if the oracle-best loss weight differs meaningfully across held-out sites in the dense grid. Interpolated models must stay within seed-to-seed variation of directly trained models at matched weights along the whole path, and the loss barrier should be small for shared-checkpoint endpoints. The label-free selector must recover most of the oracle's per-site gain over the single source-tuned weight. Success means improving Euler characteristic difference or HD95 on out-of-domain sites without lowering Dice by more than seed-level noise. Finally, the boundary study reports whether HeteroZ or GAMF alignment lowers the barrier for independently configured nnU-Nets enough to make the dial usable without a shared checkpoint. We expect it will not fully do so, and a negative answer here is useful: it would tell multi-hospital consortia that distributing one shared starting checkpoint is the price of post-hoc loss control.