Medical segmentation models are usually trained with a hybrid loss. Cross-entropy, a per-pixel classification penalty, gives stable early gradients. Dice loss, an overlap score normalized by region size, keeps a tiny structure from being swamped by a large background. Dice makes every class count equally regardless of size, but that guarantee stops at the class boundary. Inside one class, a common appearance (say, typical myocardium in mid-ventricular slices) and a rare one (thin apical myocardium, or a pathological variant) are pooled into one overlap score. The rare appearance contributes almost nothing. A separate line of work, multi-prototype classifiers, already splits each class into several learned sub-types: a pixel is labeled by its nearest of several "prototype" vectors per class rather than by a single class vector. These sub-types are trained only with cross-entropy, contrastive or distance terms, and are then summed back into class probabilities before any overlap loss is computed.
We propose to compute Dice at the level of these sub-types. Each prototype defines a latent sub-mask inside the ground-truth mask of its class, and an overlap term on each sub-mask gives every visual mode the same size-invariant weight that Dice already gives every class. Restricting this term to foreground pixels avoids multiplying the background false-positive bias that plagues soft overlap losses. The term also keeps rare prototypes alive without forcing all prototypes to equal size, which current regularizers do.
Core Claim
A sub-type-level region loss, computed only inside each class's annotated region on the multi-prototype head's own assignments, will improve segmentation of rare within-class appearances. It will do so without hurting average Dice, and the gain will not be matched by either half alone: neither a multi-prototype head with ordinary class-level cross-entropy plus Dice, nor a single-prototype network with appearance-based pixel reweighting.
Where This Sits in Prior Work
Three multi-prototype segmentation methods supply the representation and motivate the gap, because none applies a region loss per sub-type.
Rethinking Semantic Segmentation: A Prototype View (ProtoSeg) keeps several data-driven prototypes per class. It assigns pixels to them with balanced optimal-transport clustering (Sinkhorn-Knopp) and trains them with cross-entropy plus contrastive and distance losses.
Center-guided Classifier for Semantic Segmentation of Remote Sensing Images (CenterSeg) treats each class's sub-prototypes as a subspace and adds prototype-diversity and feature-to-prototype terms.
Multi-Prototype-based Embedding Refinement for Medical Image Segmentation fixes three K-means-initialized, momentum-updated prototypes per class. It enforces consistency between this prototype classifier and a linear classifier.
We reuse their machinery, namely nearest-prototype assignment with moving-average updates, and change only the supervision.
The closest method is Diversified Multi-prototype Representation for Semi-supervised Segmentation. It predicts more sub-classes than classes, aggregates them, and regularizes sub-class usage with mutual-information maximization and an orthogonality penalty. Mutual-information maximization explicitly balances how often each prototype is used across the dataset. That is the wrong prior when a sub-type is genuinely rare, because it pushes the rare mode to absorb common pixels or to merge away. Our term protects a rare mode by scoring its overlap relative to its own size, not by inflating its share.
Two loss papers constrain the design.
A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation shows a training-time distortion. Summing small soft probabilities over a huge background creates a false-positive penalty that does not exist after thresholding, and the penalty grows with the absolute pixel gap between classes. Splitting each class into $K$ sub-channels multiplies the number of channels that collect this residue, so a naive per-sub-channel Dice over the whole image would make the bias worse. That paper also treats within-class imbalance only as easy-interior versus hard-boundary difficulty, not as distinct appearance modes.
Segmentation Loss Odyssey documents that Dice gradients are erratic early in training, when predicted and true regions barely overlap. Online clustering run on those same early features is fragile, which shapes our schedule.
Understanding Imbalanced Semantic Segmentation Through Neural Collapse (CeCo) pulls each class toward a single maximally separated center. It is a deliberate counterpoint: it argues that one center per class is the right target under imbalance. Our experiments include it so that this contradiction is tested rather than ignored.
The Loss
Let $\pi_{ik}$ be the softmax probability that pixel $i$ belongs to sub-type $k$, taken over all sub-channels of all classes, so that class probability is $p_{ic}=\sum_{k\in c}\pi_{ik}$. Let $\Omega_c$ be the set of pixels annotated as class $c$. Let $a_{ik}\in\{0,1\}$ assign each pixel in $\Omega_c$ to its nearest moving-average prototype of class $c$; this assignment is treated as a fixed target, with no gradient flowing through it. The mode-balanced Dice is
$$L_{\text{mode}}=\frac{1}{|\mathcal{A}|}\sum_{c}\sum_{k\in c,\ k\in\mathcal{A}}\left(1-\frac{2\sum_{i\in\Omega_c}\pi_{ik}a_{ik}}{\sum_{i\in\Omega_c}\pi_{ik}+\sum_{i\in\Omega_c}a_{ik}}\right),$$
where $\mathcal{A}$ is the set of sub-types with at least one assigned pixel in the batch.
The full objective is $L=L_{\text{CE}}(p)+L_{\text{Dice}}(p)+\lambda L_{\text{mode}}$, with the two class-level terms computed exactly as in a standard hybrid loss.
Each design choice has a reason. Because the sums run only over $\Omega_c$, background pixels never enter the denominator, so the soft-versus-hard background bias is handled once at class level rather than amplified $K$-fold. Because each sub-type's score is normalized by its own sub-mask size, a mode covering two percent of a class's pixels carries the same weight as one covering eighty percent. Because empty sub-types are skipped, the term never forces a dead prototype to exist. Instead, the per-mode Dice and the occupancy become a diagnostic of prototype health that the static-count methods lack.
The mode term is switched on only after the assignment churn stabilizes, measured as the fraction of foreground pixels changing prototype between epochs. This addresses the early-instability interaction between Dice gradients and clustering.
Research Program
We will build the head on a 3D U-Net, following the multi-prototype heads above, and implement all region and compound losses from the SegLoss library released with Segmentation Loss Odyssey.
The analysis has three parts.
The first part is a factorial study of head and loss. The head is either a single prototype or multiple prototypes. The loss is either the class-level hybrid, the class-level hybrid with appearance-cluster pixel reweighting, or the class-level hybrid plus mode-balanced Dice.
The second part is a controlled test with known modes. LoveDA contains both rural and urban scenes, so we will train on mixtures in which urban tiles make up 5, 15 or 50 percent of the data. Ground-truth mode identity is then known. We can check whether prototypes align with it and whether the rare mode is protected.
The third part measures the background-bias claim directly. We compare our foreground-restricted term with a naive whole-image per-sub-channel Dice. On each, we record the training-time soft false-positive mass against the false positives present at inference, as $K$ grows from 1 to 8.
Baselines
Naive reference. A single-prototype 3D U-Net trained with cross-entropy plus Dice, the standard compound recipe. It is the do-nothing reference. On WMH, Wu et al. report 82.3% Dice with such a compound loss plus pretraining and deep supervision.
Strongest representation competitor. The multi-prototype head of Diversified Multi-prototype Representation for Semi-supervised Segmentation, with its mutual-information and orthogonality regularizers. It is the closest method and the one whose usage-balancing our term replaces. It reports 72.44% mean 3D Dice on ACDC with one labeled scan (82.10% with Mean Teacher) and 73.38% on PROMISE12 with three labeled scans. For the semi-supervised cardiac setting we also report against Multi-Prototype-based Embedding Refinement, which reports 88.82% Dice and 80.0% Jaccard on LA with 5% labels.
Strongest loss competitor. The Balance Loss of A Unified Loss for Handling Inter-Class and Intra-Class Imbalance, which reports 66.8% kidney-tumor Dice on KiTS19 against 59.8% for cross-entropy. It is the best existing within-class treatment on the loss side.
Other comparisons. On LoveDA, CenterSeg's reported 52.3% mIoU over a 51.2% ConvNeXt baseline anchors the remote-sensing comparison. CeCo with Dice is included as the single-center alternative.
Fairness and the beating criterion. All methods share backbone, crops, augmentation, epochs and the same hyperparameter search budget, including $K$ and $\lambda$ for us and loss weights for competitors. Labeled and unlabeled splits are identical, and we use three seeds. Beating a baseline means a gain in worst-stratum Dice (defined under Proposed Experiments) that exceeds seed variance, with mean Dice no lower than the competitor's. Mean-Dice gains alone do not count.
Key ablation. The decisive ablation is the multi-prototype head with only class-level cross-entropy plus Dice. It isolates the mode term.
Why not just…?
Why not just put a multi-prototype head on the network and keep ordinary cross-entropy plus Dice on the summed class probabilities? This is the most direct threat, since aggregation already makes class Dice available. Class Dice cannot distinguish missing a rare mode from shaving the edge of a common one; both remove the same number of pixels. The mode-usage collapse that the Diversified Multi-prototype paper needed extra regularizers to prevent should therefore persist. The factorial study settles this. If this ablation matches our method on worst-stratum Dice, the mode term adds nothing, and we will report that.
Why not just reweight hard pixels with Focal, TopK or Balance Loss on a single-prototype network? Rare modes are often hard, so difficulty weighting may partly recover them. Difficulty is a per-pixel continuum, however, while a rare mode can be confidently wrong, and the Unified Loss paper notes that TopK-style discarding destabilizes training. We compare against Balance Loss directly. We also stratify its errors by mode on LoveDA, where modes are known, to see whether it protects the rare mode or only boundaries.
Why not just cluster features of a single-prototype network and reweight pixels by inverse cluster size? This captures the loss idea without a multi-prototype head, and it could plausibly match us. If it does, the contribution is the loss and the head is unnecessary. The factorial study includes exactly this cell with the same clustering. The large head-only gains reported by the Diversified Multi-prototype paper (52.33% to 73.38% on PROMISE12) suggest the head matters in low-label regimes, but whether head and loss interact is precisely what we measure.
Proposed Experiments
Datasets and regimes. The medical benchmarks are ACDC at 1 labeled scan and at 10% labels, PROMISE12 at 3 scans, LA at 5%, KiTS19 tumor, and WMH with external testing on MACC. The controlled-mode benchmark is the imbalanced LoveDA mixtures.
Metrics. Beyond mean Dice, Jaccard and 95th-percentile Hausdorff distance, the central metric is worst-stratum Dice over strata that approximate modes without sub-labels. On ACDC these are apical, mid and basal slices and the five diagnostic groups. On KiTS19 they are tumor-size tertiles. On LoveDA they are the true rural and urban modes.
Decisive checks. First, on LoveDA at 5% urban, the method must raise urban mIoU over the multi-prototype-plus-class-Dice ablation. Its prototypes should also align with the true modes better than the usage-balancing regularizer's prototypes, measured by mutual information between prototype assignment and mode label. Second, on KiTS19 and ACDC, worst-stratum Dice must exceed both Balance Loss and the cluster-reweighted single head. Third, the foreground-restricted term must show flat training-to-inference false-positive mismatch as $K$ grows, while the naive whole-image variant's mismatch grows roughly linearly in $K$. That linear growth is expected because each added sub-channel collects its own background residue.
If the first check fails, prototypes are not tracking real modes, and the premise is wrong. If the second fails only against the cluster-reweighted single head, we will reframe the result as a loss contribution independent of the head.
References
- Rethinking Semantic Segmentation: A Prototype View
- Center-guided Classifier for Semantic Segmentation of Remote Sensing Images
- Multi-Prototype-based Embedding Refinement for Medical Image Segmentation
- Diversified Multi-prototype Representation for Semi-supervised Segmentation
- A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation
- Segmentation Loss Odyssey
- Understanding Imbalanced Semantic Segmentation Through Neural Collapse
- Deeply supervised network for white matter hyperintensities segmentation with transfer learning