A common way to train a medical segmentation model is with a small set of labeled scans from one hospital and many unlabeled scans from several others. Current methods pool all the unlabeled hospitals into one averaged "be consistent with yourself" loss. That average lets the model do well on the hospitals that look like the labeled one and quietly fail on the hospital whose scanner differs most. That hospital is usually the best stand-in for the next, unseen scanner. We propose replacing the average with a smoothed maximum over hospitals. The smoothing uses log-sum-exp, a differentiable "soft max" whose temperature slides it between a plain average and the true worst case. The most important design choice is what goes inside that maximum. On unlabeled data the raw consistency loss is a poor measure of failure: a model that is confidently wrong looks consistent. So we take the soft maximum over each hospital's excess consistency loss above a hospital-specific floor. This is an unlabeled analogue of regret, the shortfall relative to the best loss attainable on that hospital.
The claim can be stated in two sentences. Semi-supervised segmentation under multi-site shift should optimize a temperature-controlled soft worst case over unlabeled sites, so that training effort flows toward the site the model handles worst. That effort only lands in the right place if the per-site signal is a regret-style quantity, because raw self-consistency rewards confident errors. The method is a drop-in replacement for the unlabeled-loss aggregation in existing pipelines. At one end of its temperature range it recovers the current uniform average exactly, which makes the comparison clean.
What exists and where this sits
Three lines of work meet here. The first is mixed-domain semi-supervised segmentation, where one site is labeled and several are not. MiDSS builds intermediate domains by copy-paste and Fourier amplitude mixing, then cross-supervises students with dual pseudo-labels. DAC co-trains two models with shared feature targets and different self-supervised side tasks. A&D GenericSSL separates labeled and unlabeled training into different decoders to counter "supervised dominance", its term for the labeled loss crowding out the unlabeled one. All three motivate this proposal and supply its testbed. All three also combine the unlabeled sites with fixed or heuristic weights, so none of them can redirect effort toward the site that is failing. We keep their augmentation and pseudo-labeling machinery and change only how the per-site unlabeled losses are combined.
The second line is worst-group optimization with unlabeled data. Semi-supervised Group DRO optimizes the maximum over groups of regret rather than raw loss. For groups without labels it uses a proxy regret, measured against pseudo-labels from the best hypothesis on data-rich groups. Its reason is that raw worst-case loss over-weights groups that are intrinsically hard or noisy. That paper supplies our key conceptual ingredient. However, it uses hard max and is tested only on linear or convex models, synthetic regression and Colored MNIST. We carry proxy regret into dense deep segmentation, where the hard max has to be smoothed and the regret floor has to be estimated cheaply.
The third line is log-sum-exp smoothing of max and min operators. STCH-Set smooths nested Tchebycheff max/min objectives so that stochastic gradient descent converges, and anneals the temperature from smooth to sharp. FOOPS uses a log-sum-exp merit function to steer multi-objective training toward preferred trade-offs. Blanchet et al. use soft-max smoothing to make a distributionally robust minimax solvable by SGD. These papers supply the method: smoothing, annealing, and the known bias of order $\log M/\beta$. None of them has been applied to per-site unsupervised loss streams.
We are deliberately not pursuing two framings. We do not cast the labeled-versus-unlabeled balance as a FOOPS-style preference problem. With only two objectives, a tuned scalar weight is hard to beat, and the file offers no evidence that this trade-off front is non-convex in segmentation. The second framing is "LSE over unlabeled consistency losses" by itself. Its gradient is a softmax weighting of sites, which is essentially the online exponentiated-gradient update of standard Group DRO. Smoothing alone is therefore not new. What has not been done is the smoothed regret over unlabeled sites in dense segmentation, together with a test of whether that is what moves unseen-vendor performance.
Core Claim
Let $U_m(\theta)$ be the unlabeled consistency or pseudo-label loss on site $m$, for $m = 1,\dots,M$, as computed by the host method such as MiDSS's symmetric guidance loss. Let $b_m$ be a per-site floor. Let $L_{\text{sup}}$ be the labeled loss, $\lambda$ the usual unlabeled weight, and $\beta>0$ an inverse temperature. We train
$$ \mathcal{L}(\theta) = L_{\text{sup}}(\theta) + \lambda \cdot \frac{1}{\beta}\log\Big(\frac{1}{M}\sum_{m=1}^{M} e^{\beta\,(U_m(\theta) - b_m)}\Big). $$
As $\beta \to 0$ the unlabeled term becomes the plain average of $U_m - b_m$, which is today's practice up to a constant. As $\beta \to \infty$ it becomes the hard worst site. Its value always lies between the average and the maximum, and it falls short of the maximum by at most $(\log M)/\beta$, which follows from bounding the sum by $M$ times its largest term. The gradient with respect to $\theta$ is a softmax-weighted sum of per-site gradients, with weights $w_m \propto e^{\beta(U_m - b_m)}$. This is the soft gradient routing that keeps every site learning while concentrating effort on the worst one.
The floor $b_m$ is what makes the objective a regret. We estimate it as the consistency loss of the same network after a cheap, label-free adaptation to site $m$ alone. Concretely, we re-estimate the normalization statistics on site $m$'s unlabeled scans and run a short self-training pass, refreshing the estimate every few epochs. This approximates "the best this architecture can do on site $m$", which corresponds to the role of the best-group hypothesis in Semi-supervised Group DRO. The resulting prediction is concrete. Raw $U_m$ will rank sites poorly against their true error, because overconfident pseudo-labels make a badly shifted site look consistent, as CalibrateMix and Ouyang et al. document for overconfidence under shift. The excess $U_m - b_m$ should rank them correctly.
Research Program
The first component handles minibatch noise. The log of a sum of noisy minibatch estimates is biased, which is a known open problem for log-sum-exp under SGD. We therefore sample a fixed number of scans per site at every step. The weights $w_m$ are computed from exponential moving averages of $U_m - b_m$, while gradients flow through the current minibatch losses. We also report the naive in-batch variant to measure how much the bias matters in practice.
The second component is a temperature schedule. Following STCH-Set's continuation strategy, we anneal $\beta$ from a value where the $(\log M)/\beta$ gap is large, which starts close to uniform, toward a value where that gap falls below the typical spread of site losses. With $M$ between 2 and 4 unlabeled sites, the gap at a fixed $\beta$ is small, so we expect the schedule to matter less here than in many-objective settings. We test that expectation rather than assume it.
The third component is a diagnostic study that works independently of any accuracy gains. We hold out labels on the "unlabeled" training sites and use them only for analysis. Throughout training we measure the rank correlation between each site's true Dice error and its weight under three signals: raw $U_m$, excess $U_m - b_m$, and uniform. This tests the mechanism directly. If raw consistency already tracks true error, the regret floor adds nothing, and we will say so.
Baselines
The naive reference point is the host method run unchanged with its uniform average over unlabeled sites. We also run a variant with site-balanced sampling, which removes the confound that some sites simply contribute more scans. Any gain we report has to exceed the balanced-sampling variant, not just the vanilla one.
The strongest existing method in this exact setting is MiDSS. On the multi-center Fundus benchmark it reports an average Dice of 87.66%, against a fully labeled upper bound of 88.46%. It also reports a 13.57% Dice improvement over prior methods on the multi-site Prostate benchmark. The file has no reported per-vendor M&Ms numbers for MiDSS (no reported result found on Lacuna for MiDSS per-vendor Dice on M&Ms), so we will reproduce it ourselves. We implement our objective inside the MiDSS code path, replacing only the aggregation step. Both arms share the same backbone, augmentations, training epochs, labeled split and seeds. The $\lambda$ and learning-rate search uses a grid of the same size for both arms, selected on the labeled site's validation set only, because no labels from target sites may be used for tuning. For a second host method we use DAC, which reports 76.35% optic-cup Dice with 10% labeled source data. We use it to check that the effect is not specific to MiDSS.
Fundus is already within 0.8 Dice points of its upper bound, so average Dice has little headroom. Our primary metric is therefore worst-site Dice, together with Dice on the held-out site. We count it as beating MiDSS if worst-site Dice improves by at least one point, averaged over three seeds and outside seed variation. We also require that average Dice is no more than 0.3 points lower and that Hausdorff distance does not worsen.
The direct competitor for the mechanism is online Group DRO applied to the unlabeled site losses. It uses exponentiated-gradient weights on raw $U_m$, with its step size tuned on the same budget. We also run a hard-max variant that updates only the worst site at each step. Three ablations isolate the key ingredient: smoothed max over raw $U_m$ versus over $U_m - b_m$, fixed versus annealed $\beta$, and in-batch versus moving-average weights. The prediction that matters most is that regret beats raw loss under both soft and hard maximization.
Why not just…?
Why not just use heavy intensity augmentation, nnU-Net style, and keep the average? Augmentation is the strongest simple remedy, and the M&Ms challenge reports that top methods cut the unseen-vendor drop from about 6% to 1.5%. That remaining 1.5%, concentrated on the unseen Canon vendor and on the base and apex slices, is precisely our target. AMOS reports drops of over 20% on unseen MRI scanners despite standard pipelines. Augmentation and our objective are also not exclusive: the MiDSS host already applies Fourier amplitude mixing, so our arms keep all of it. If the gain disappears once augmentation is strong, the experiments will show that, and that would be a legitimate negative result.
Why not just use standard Group DRO on the per-site losses? This is the real threat. A log-sum-exp gradient with moving-average weights is mathematically close to online Group DRO, so smoothing by itself is not our contribution. The proposal stands or falls on the regret floor. If Group DRO on raw $U_m$ matches our method, or if the diagnostic shows that raw consistency already ranks sites by true error, then the regret idea is refuted, and we will report it as such. Semi-supervised Group DRO gives the theoretical reason to expect raw worst-case loss to misdirect effort, but only for linear models. Whether that holds for deep segmentation is the open question.
Why not just tune a static weight per unlabeled site? With no labels on those sites, there is no principled validation signal for choosing per-site weights. Selecting them on the labeled site's validation set tends to favor sites that resemble it, which is the supervised dominance that A&D describes. To make this baseline as strong as possible, we include an oracle version tuned with held-out target labels as an upper reference. If our label-free method approaches the oracle, the adaptive weighting is doing useful work.
Proposed Experiments
We use three benchmarks. On M&Ms, labeled data comes from one vendor (Siemens), unlabeled data from Philips and GE, and the never-seen Canon vendor serves as the held-out test domain; we report LV, RV and MYO Dice, Hausdorff distance, and a separate score on the base and apex slices. On Fundus, we follow MiDSS and DAC's protocols, rotate which domain is labeled, and report optic disc and optic cup Dice per site. On the multi-site Prostate benchmark we use leave-one-site-out evaluation. Calibration is measured by expected calibration error, ECE, on the held-out site, because a method that only shifts the trade-off toward confident errors should show up there.
Two outcomes are decisive. The first is held-out-vendor Dice on M&Ms relative to MiDSS under matched tuning, which tests whether optimizing the training-site worst case transfers to an unseen scanner. The second is the regret-versus-raw ablation under both soft and hard maximization, read together with the rank-correlation diagnostic. The idea is supported only if regret wins both comparisons and its weights track true per-site error better than raw consistency does.
References
- Constructing and Exploring Intermediate Domains in Mixed Domain Semi-supervised Medical Image Segmentation
- Dual-supervised Asymmetric Co-training for Semi-supervised Medical Domain Generalization
- Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation
- Semi-supervised Group DRO: Combating Sparsity with Unlabeled Data
- Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimization
- Efficient First-Order Optimization on the Pareto Set for Multi-Objective Learning under Preference Guidance
- Data-Driven Optimal Transport Cost Selection For Distributionally Robust Optimization
- On the Calibration of Image Semi-Supervised Learning Models
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation