Medical image segmentation models are compared by a single row of numbers: one overlap score (Dice, the fraction of agreement between predicted and expert masks) and one boundary score (Hausdorff distance, roughly the worst-case gap between predicted and true outlines). That row comes from one arbitrary operating point. In practice the same trained network can be pushed along a curve by cheap, training-free choices: the probability cutoff for calling a voxel "organ," whether small disconnected blobs are deleted, and how far the outline is grown or shrunk. Each choice trades overlap against boundary error. When a paper reports that a new loss or a new edge branch lowers Hausdorff distance, we usually cannot tell whether the network learned something new or simply landed on a different point of a curve the old model could already reach with post-processing.
This proposal borrows the main lesson of multi-objective optimization evaluation, that an algorithm should be judged by the set of trade-offs it can offer and not by one point from that set, and applies it to segmentation. Each trained model is treated as the generator of an attainable set: all (overlap, boundary, false-positive) outcomes reachable by post-hoc knobs that are tuned on validation data and scored on test data. Models are then compared by dominance and by a hypervolume restricted to clinically acceptable regions. The decisive empirical question is concrete. On standard benchmarks, how much of the published improvement from boundary-aware training survives once the baseline is allowed the same post-processing freedom?
Core Claim
We claim that a substantial and measurable fraction of reported boundary improvements in medical segmentation are movements along a baseline's attainable trade-off curve rather than outward shifts of it. We also claim that set-based comparison over post-processing-reachable outcomes changes method rankings on public benchmarks. The motivating case is Regional Hausdorff Distance Losses for Medical Image Segmentation. It builds a differentiable soft-erosion distance transform so Hausdorff distance can be trained directly. On MSD Pancreas it reports HD falling from 25.14 to 16.94 and Dice moving from 0.77 to 0.82 relative to a Dice-loss baseline, both evaluated at a single fixed operating point. Hausdorff distance is a maximum, so one spurious blob far from the pancreas dominates it, and deleting small components is exactly the kind of fix that post-processing provides for free. We do not know whether the trained loss beats a post-processed Dice model. If the claim holds, part of the boundary-loss literature needs re-evaluation. If it fails, boundary losses get stronger evidence than they currently have.
What is new here relative to the closest work is the object being evaluated. Metrics reloaded built an expert-consensus process ("problem fingerprinting") for choosing complementary overlap, boundary and clinical-cost metrics. It motivates our metric choice. It is an a posteriori metric-selection guide, though, and still scores each model at one point; it does not ask what trade-offs a model could reach. Quality Evaluation of Solution Sets in Multiobjective Optimisation: A Survey supplies the method: set-quality indicators such as hypervolume (the volume of objective space a set dominates up to a reference point, which respects dominance) and IGD+ (distance from a reference front). It also supplies the warning that no single indicator suffices. The critique by Chen, Li and Yao in How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? shapes the design. They show that comparing mean objective values and using global indicators that ignore where decision makers actually operate produce misleading rankings. We answer with a preference-restricted hypervolume whose region is set by clinical tolerances, in place of a global one.
Research Program
The first component is a shared training testbed. Within one nnU-Net-style codebase (nnU-Net) we train a small set of variants with matched data, splits, epochs and augmentation. The variants are Dice plus cross-entropy, the symmetric averaged Hausdorff loss from the regional Hausdorff paper, an auxiliary edge-decoder branch in the style of the edge path in DB-MSMUNet, and an iterative mask-refinement decoder in the style of MDNet. We reimplement rather than use released checkpoints so that only the ingredient under test differs between variants.
The second component is the attainable-set construction. Each trained model gets the same post-processing grid of roughly 150 configurations. The grid crosses four knobs: probability threshold, minimum connected-component volume, keep-largest-component on or off, and a signed boundary offset applied to the thresholded probability map's distance transform. For each configuration we compute mean Dice, mean HD95 and a false-positive component rate over the validation fold. The non-dominated configurations form the model's validation front. We then evaluate exactly those configurations on held-out test cases, which gives the test front. Selecting on validation and reporting on test keeps the post-processing freedom from becoming test-set tuning.
The third component is comparison. The main score is hypervolume inside a clinical box. Its reference point sits at tolerances fixed in advance from the problem-fingerprint axes of Metrics Reloaded, for example a minimum acceptable Dice and a maximum acceptable HD95 per organ, so that trade-off regions no clinician would use earn nothing. This responds directly to the preference-misalignment finding of Chen et al. The secondary scores are pairwise set dominance and a per-patient dominance rate, which checks that front-level conclusions are not an artifact of averaging across patients. The true front is unknown, the "ground truth deficit" in the file. For IGD+ we therefore build a reference set from the merged archive of all models' validation outcomes, made uniform with the Reference Set Generator. That method clusters disconnected pieces, fills gaps by triangulation and downsamples by k-means. Hypervolume, which needs no reference front, remains the headline. Uncertainty comes from a patient-level bootstrap over the whole pipeline, including front selection.
Baselines
The naive reference point is the current protocol. Each model is scored at threshold 0.5 with no post-processing, using mean Dice and mean HD95 as in the original papers. Every other comparison is measured against the rankings this produces.
The strongest existing method on the question "does training on boundary distance beat overlap training?" is the symmetric averaged Hausdorff loss of Regional Hausdorff Distance Losses for Medical Image Segmentation. It is the only work in the file that optimizes Hausdorff distance end to end, and it reports on MSD Pancreas an HD of 16.94 and Dice of 0.82 against 25.14 and 0.77 for its Dice-loss baseline. We first reproduce that single-point gap in our codebase, to confirm our implementation matches, before granting either model post-processing. Beating it means one of two things. A post-processed Dice-loss model may reach a test-front point that is no worse in both Dice and HD95 than the boundary-loss model's default point, which shows the gain is reachable without the loss. Alternatively, the boundary-loss model's clinical-box hypervolume may exceed the Dice model's with a bootstrap interval excluding zero, which shows a genuine front shift. Either outcome is informative. For further single-point context, MDNet reports DSC 0.9507 and HD 2.26 on MSD Spleen and DSC 0.9383 and HD 3.79 on LiTS. The cascaded network of Wang et al. reports BraTS 2017 validation Dice of 0.7859, 0.9050 and 0.8378 for enhancing, whole and core tumor. Those protocols differ from ours, so these are anchors and not targets. We found no reported result on Lacuna for hypervolume or any set-based indicator on any medical segmentation benchmark.
The strongest practical competitor is nnU-Net's own validation-selected post-processing, which picks the one component-removal setting that maximizes Dice. Every variant receives it with the same validation data. It is the right control because it already captures the "just clean up the blobs" intuition at a single point. The key ablation removes one knob family at a time, threshold, component filtering or boundary offset. This isolates which kind of post-hoc freedom drives any ranking change and shows whether the effect is just "removing blobs fixes HD."
Why not just…?
Why not just report Dice and HD95 side by side, as Metrics Reloaded recommends? Side-by-side reporting is necessary but leaves the comparison undecided whenever one model wins on Dice and the other on HD95. It also says nothing about whether the loser could have reached the winner's point. The headline experiment shows whether this matters. If attainable-set comparison never reverses the side-by-side conclusion on any of our tasks, we will report that the simpler practice is adequate.
Why not just apply nnU-Net's post-processing to every model and compare single points? This is the real threat. If nnU-Net's single Dice-optimal setting already closes most of the boundary-loss gap, the full front adds little. The ablation that compares the nnU-Net-selected point against the full validation-selected front settles it. The front matters only if a clinician who prefers a lower-HD95 point would choose a different model than the Dice-optimal point does, and we count how often that happens across tasks and tolerance boxes.
Why not just scalarize, ranking by Dice minus λ times HD95 and sweeping λ? For each fixed λ this is a cheap, defensible scalar. Linear scalarization recovers only the convex part of a front, as the indicator survey notes, and the comparison of hypervolume-aware search with fixed scalarization in A study of EHVI vs fixed scalarization for molecule design found a hypervolume of 0.4022 ± 0.0661 versus 0.3492 ± 0.0190 on Fexofenadine. That evidence comes from search, not evaluation, and from molecules, so it only suggests a gap. We test the question directly by checking whether the λ-sweep ranking matches the clinical-box hypervolume ranking. If the post-processing fronts turn out to be nearly convex, the rankings will agree, and we will recommend the simpler sweep.
Proposed Experiments
We run four MSD tasks, Pancreas, Spleen, Liver and Brain Tumour (MSD), using five-fold cross-validation for training and front selection plus a held-out test split. NIH Pancreas serves as a second pancreas cohort to check that conclusions transfer. The first decisive check is the pancreas reproduction-then-relaxation. We confirm the single-point gap between the boundary loss and Dice loss, then measure what fraction of the HD95 gap the Dice model's test front closes at matched Dice. The second check is a ranking audit. Across the four training variants and all tasks, we compute Kendall's rank correlation between the default single-point ranking and the clinical-box hypervolume ranking, with bootstrap intervals, and count strict reversals. The third check is robustness. We vary the clinical box within plausible tolerance ranges and confirm that any reversal is not an artifact of one reference point. We also confirm that the per-patient dominance rate agrees in direction with front-level hypervolume.
Before running anything, we commit to the following reading of the results. If post-processing closes less than a quarter of the boundary-loss gap on pancreas and rankings agree everywhere, the current single-point protocol is vindicated for these methods and we report that. If it closes most of the gap, or rankings reverse on two or more tasks, segmentation papers should report attainable fronts. We would then release the grid, the clinical-box definitions and the scoring code as a drop-in addition to the existing metric implementations.
References
- Regional Hausdorff Distance Losses for Medical Image Segmentation
- Metrics reloaded: recommendations for image analysis validation
- Quality Evaluation of Solution Sets in Multiobjective Optimisation: A Survey
- How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance
- nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation
- DB-MSMUNet: Dual Branch Multi-Scale Mamba UNet for Pancreatic CT Scans Segmentation
- MDNet: Multi-Decoder Network for Abdominal CT Organs Segmentation
- Reference Set Generator: A Method for Pareto Front Approximation and Reference Set Generation
- Automatic Brain Tumor Segmentation Using Cascaded Anisotropic Convolutional Neural Networks
- A study of EHVI vs fixed scalarization for molecule design
- A large annotated medical image dataset for the development and evaluation of segmentation algorithms