Segmentation networks for medical images are usually trained with a hybrid loss, meaning a weighted sum of pixel-wise cross-entropy and an overlap score such as Dice. Dice handles tiny targets well, but it pushes output probabilities to 0 or 1. As a result, the network's own confidence is a poor guide to when it should defer a scan, or a region of a scan, to a radiologist. We propose to stop asking the softmax for its confidence. Instead, we freeze the trained segmentation network and fit a separate linear "error head" on its internal features. This head predicts, for each candidate region of the predicted mask, the fraction of voxels that are wrong. It is fit in one step by regularized least squares. Because squared error on a probability target is the Brier score, a proper scoring rule, the head is pushed toward calibrated error rates and not only toward a good ranking.
The closed form does more than save training time. It buys three things that a gradient-trained auxiliary network does not have. First, exact leave-one-subject-out predictions are cheap, so the deferral threshold can be set with jackknife-style guarantees on subject-level error. Second, the head can be refit in seconds on a handful of labeled scans from a new hospital, with a proximal term that shrinks it toward the source solution. Third, the fit has no learning rate, schedule or early-stopping choices to tune. The bet is simple and falsifiable: even when the hybrid loss destroys error information in the output probabilities, that information survives in the penultimate features, and a linear solve can read it out.
Core Claim
Our central empirical claim concerns the weighting between cross-entropy and Dice in the hybrid loss. As Dice gets more weight, the ability of softmax-based uncertainty to separate errors from correct predictions should degrade. We measure that ability with AURC, the area under the risk-coverage curve, where lower is better. A closed-form error head on frozen features should stay roughly flat across the same weighting sweep. Our methodological claim is that one regularized least-squares fit, with a proximal anchor and exact leave-one-out residuals, yields a deferral rule that is ranked well and calibrated at the subject level. This is the trade-off the literature currently presents as a choice.
That literature splits into two stances. Uncertainty-Aware Training of Neural Networks for Selective Medical Image Segmentation proves that selective risk depends on ranking errors above correct predictions, not on calibration. Revisiting Reweighted Risk for Calibration: AURC, Focal Loss, and Inverse Focal Loss argues the reverse: minimizing calibration error is formally tied to selective risk. Our head targets the conditional error probability itself. If it is estimated well, it gives both a good ranking and well-calibrated error rates, and our experiments measure each property separately.
Relation to Prior Work
Two papers supply the method. Superfast-Trainable Multi-Class Probabilistic Classifier by Least-Squares Posterior Fitting fits class posteriors with kernel models under squared loss and solves them analytically. It matches kernel logistic regression at about 100 times the speed. Improving the Accuracy of Least-Squares Probabilistic Classifiers shows that clipping negative outputs after the fit beats constraining parameters during it. We reuse both ideas, but with one target: the probability that a region is wrong, rather than class membership. Closed-Form Last Layer Optimization solves a network's final layer by ridge regression and adds a proximal term that behaves like a Kalman update. It also reports that squared loss trails cross-entropy on ImageNet classification.
That negative result constrains our design. We never use squared loss to learn the segmentation itself. The backbone keeps its hybrid loss, and squared loss is used only to regress a bounded scalar error rate. For that task the many-class weakness Galashov et al. identify does not arise.
Three further papers motivate the gap. Calibrating the Dice loss to handle neural network overconfidence for biomedical image segmentation shows that Dice drives overconfidence. Its DSC++ loss fixes this at training time, but it does not build any deferral rule. Post Training Uncertainty Calibration Of Deep Networks For Medical Image Segmentation shows that post-hoc scaling lowers population calibration error while per-subject calibration variance "doesn't change much." That per-subject variance is the failure our leave-one-subject-out and few-scan refitting are meant to address. Calibrated Selective Classification trains a selector by gradient descent under simulated shift for whole-image classification. Our selector instead works on regions inside a segmentation and is solved analytically.
Research Program
Each segmentation backbone is a standard U-Net-style network trained with $\alpha\,\mathcal{L}_{\text{CE}} + \beta\,\mathcal{L}_{\text{Dice}}$. We sweep the ratio $\beta/\alpha$ over several values, and we also train a DSC++ variant.
The unit of deferral is a region rather than a voxel. Abstaining voxel by voxel yields fragmented masks, while abstaining on whole volumes throws away mostly-correct work. We therefore partition each predicted mask into regions using three kinds of piece: connected components, boundary bands of fixed width around each component, and supervoxels in large structures.
For each region $r$ we build a feature vector $\phi_r$. It pools the frozen penultimate features, inside and on the boundary of the region, and is augmented with random Fourier features so that the linear fit can capture nonlinear relations. The regression target $e_r \in [0,1]$ is the fraction of voxels in the region whose predicted label disagrees with ground truth. Targets are computed on a calibration set of subjects that the backbone never saw during training.
The head weights $w$ are fit as
$$ w = \left(\Phi^\top \Phi + (\lambda + \rho) I\right)^{-1}\left(\Phi^\top e + \rho\, w_0\right). $$
Here $\Phi$ stacks the region features, $e$ stacks their error fractions, $\lambda$ is ridge strength, and $w_0$ is a prior solution with proximal weight $\rho$. On the source data, $\rho=0$. When adapting to a new site, $w_0$ is the source head.
The Gram matrix $\Phi^\top\Phi$ is $d\times d$ with $d$ in the low thousands, and it is accumulated by streaming over regions. The cost therefore does not grow with voxel count, which sidesteps the dense-grid sorting cost that Revisiting Reweighted Risk for Calibration left unaddressed. Predictions are clipped to $[0,1]$ after fitting, following Yamada et al.
The deferral score for a region is $\hat e_r = \phi_r^\top w$. Leave-one-subject-out residuals come from the block hat-matrix identity at the cost of one extra solve per subject. We use these residuals in two ways. First, they select $\lambda$ exactly, without a held-out split. Second, they set a threshold on $\hat e_r$ that bounds the expected error on retained regions at a target level, with a jackknife-style finite-sample argument.
We will also analyze where the head draws its signal. To do this, we compare features from different backbone depths and measure how far the head's ranking quality degrades when softmax-derived inputs are removed from $\phi_r$.
Baselines
The naive reference point is the backbone's own maximum softmax probability or entropy, averaged over each region. It costs nothing and is what a deployed system would use by default. We will also apply temperature scaling to it. Temperature scaling cannot change the ranking, so it should leave AURC unchanged while improving calibration. Showing this separates the two properties cleanly.
The calibration baseline is post-hoc Platt scaling and an auxiliary calibration network trained by gradient descent, as in Post Training Uncertainty Calibration Of Deep Networks For Medical Image Segmentation. It reports BraTS 2018 population calibration error dropping from about 1.765% to about 1.028%. Both are fit on the same calibration subjects as our head. Beating this baseline means matching its population calibration error while cutting the across-subject standard deviation of calibration error by a clear margin.
The strongest existing selective-segmentation method is the end-to-end ranking loss of Uncertainty-Aware Training of Neural Networks for Selective Medical Image Segmentation. On MM-WHS it reports AURC 0.810 against a 0.936 baseline, and Dice at 95% coverage of 94.72% against 93.86%. On GlaS it reports AURC 6.031 against 6.981. We will reimplement it on our backbones with the same training data, and give its margin and weight hyperparameters a tuning budget equal in GPU-hours to all of our tuning.
Beating it means a lower AURC at both voxel and region level on MM-WHS and GlaS, with no retraining of the backbone. Retraining-free parity would already count as a meaningful win.
We include Monte Carlo dropout with 20 passes as the sampling baseline, and DSC++ with entropy as the training-time calibration baseline. We will check that our reimplemented baselines approximately reproduce the cited numbers before comparing.
The key ablation replaces the closed-form solve with the same linear head trained by Adam on identical features and targets. For the adaptation setting, this ablation uses weight decay toward $w_0$. It isolates what the closed form buys, as opposed to what the features buy. A second ablation feeds the head softmax-derived inputs only, with no backbone features, which tests whether the information really survives in the features.
Why not just…?
Why not just use softmax entropy with temperature scaling? This is the real threat to the idea. If entropy ranking already matches the head's AURC at the Dice-heavy end of the weighting sweep, the core claim is false. Yeung et al. give reason to expect saturation under Dice, because their outputs collapse toward 0 and 1. Saturated outputs cause ties and bad orderings that no monotone rescaling can repair. The loss-weighting sweep settles this directly, and we will report the result even if entropy holds up.
Why not just train a small error-prediction network by gradient descent on the calibration set? On a large calibration set this may well match our accuracy, and the Adam ablation is designed to show exactly that. The closed form's case rests on two other properties. One is exact leave-one-subject-out estimates for threshold setting. The other is stable adaptation from three to five labeled target scans, where an iteratively trained auxiliary network is prone to overfit. If the Adam ablation also matches us in the few-scan regime, we will report that the gains come from feature-based error regression, not from the closed form.
Why not just retrain with the ranking loss of Ding et al. or with DSC++? Both work, and both are included as baselines. They require retraining every deployed backbone, however, and each comes with weighting or margin hyperparameters to tune. Neither offers a mechanism for per-site refitting. If they beat our head by a wide AURC margin, we will report that retraining is worth its cost. We will also test whether stacking our head on a DSC++ backbone adds further gains.
Why not just use Monte Carlo dropout? It costs tens of forward passes per volume. BRATS-era pipelines already range from 0.3 to 100 minutes per case (The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)), and Rousseau et al. found single-pass post-hoc methods competitive with it on calibration. We report its AURC alongside wall-clock time, so readers can judge the trade-off.
Proposed Experiments
We evaluate on five benchmarks:
- MM-WHS CT, using the protocol of Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation, on whose challenge test set the best reported CT Dice is 0.91±0.09;
- GlaS, the gland segmentation benchmark, where both Ding et al. numbers are available;
- BraTS 2018, for subject-level calibration;
- KiTS19, for small tumors;
- DRIVE, for thin retinal vessels.
On KiTS19, Yeung et al. report tumor Dice of 0.232 with standard Dice loss and 0.429 with DSC++. On DRIVE they report negative log-likelihood of 0.204 and 0.041 respectively. For ISLES 2018 the file reports no number we can use (no reported result found on Lacuna for ISLES 2018 calibration values), so it serves only as a secondary check.
The metrics are:
- AURC at voxel and region level;
- Dice at 90% and 95% coverage;
- Brier score and calibration error of the predicted error rates;
- across-subject standard deviation of per-subject calibration error;
- fitting time.
The first decisive check is the loss-weighting sweep. Softmax-entropy AURC should worsen as Dice weight rises, while the head's AURC stays within a small band.
The second decisive check is subject-level. Using leave-one-subject-out thresholds, the observed error on retained regions should stay below the target for nearly all held-out subjects. Post-hoc Platt scaling should not achieve this, in line with Rousseau et al.'s finding of persistent per-subject variance.
The third check is few-scan adaptation. We will apply AugMix-style corruptions, following the shift-simulation approach of Calibrated Selective Classification. If acquisition-site labels are available in BraTS, we will also hold out a site. Proximal refitting from three to five labeled target scans should recover most of the AURC lost under shift, and it should do better than the Adam-trained head given the same scans.
If the first check fails, the proposal reduces to a fast recalibrator and we will say so. If the second or third fails relative to the Adam ablation, the closed form's contribution is limited to speed.
References
- Uncertainty-Aware Training of Neural Networks for Selective Medical Image Segmentation
- Revisiting Reweighted Risk for Calibration: AURC, Focal Loss, and Inverse Focal Loss
- Superfast-Trainable Multi-Class Probabilistic Classifier by Least-Squares Posterior Fitting
- Improving the Accuracy of Least-Squares Probabilistic Classifiers
- Closed-form Last Layer Optimization
- Calibrating the Dice Loss to Handle Neural Network Overconfidence for Biomedical Image Segmentation
- Post Training Uncertainty Calibration Of Deep Networks For Medical Image Segmentation
- Calibrated Selective Classification
- The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)
- Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation: An Open-Access Grand Challenge