Medical image segmentation, the task of outlining organs or tumors pixel by pixel, is trained on masks drawn by human experts who often disagree about where a boundary lies. Curriculum learning trains a model on easy examples first and hard ones later, and it has been applied to segmentation by softening or down-weighting boundary pixels early and sharpening them later. These curricula have a known weakness. Their schedule settings (when to switch phases, how fast to admit hard pixels, how quickly to decay smoothing) have to be retuned for every new dataset. This note argues that, in segmentation, much of that fragility has a specific cause. The curricula invent a proxy for boundary difficulty, such as blur kernels or the model's own uncertainty, and anneal toward one-hot targets that assert certainty the experts never had. When several experts have annotated each image, the data already contain a measured difficulty: the fraction of raters who agree on each pixel.
We propose to make rater agreement the curriculum's clock. Training first admits only pixels on which all raters agree. It then admits pixels of decreasing agreement, and the final target is the soft distribution of rater votes rather than a sharpened mask. The schedule advances when the model's predictions on the currently admitted pixels match the rater frequencies, not after a fixed number of epochs. The main deliverable is a controlled sensitivity study showing whether this change flattens the performance landscape over schedule settings and lets one schedule transfer across datasets. The study also tests the uncomfortable possibility that plain soft-label training with no curriculum at all does just as well.
Core Claim
The claim can be tested directly. On multi-rater segmentation benchmarks, curriculum hyperparameter sensitivity decomposes into two sources. The first is target misspecification: annealing contested pixels toward hard labels. The second is loss gradient dynamics: region losses such as Dice push predictions toward 0 or 1 regardless of true ambiguity. Removing both sources should shrink the spread of results across a grid of pacing settings, and it should shrink the loss incurred when a schedule tuned on one dataset is reused on another. Rater agreement serves as the difficulty score and calibration serves as the pacing signal. If the sensitivity survives both fixes, then sensitivity is intrinsic to curriculum pacing in this domain, and that is also worth knowing.
Two definitions are needed. For a pixel $x$ annotated by $R$ raters, $k(x)$ of whom mark it foreground, the agreement is $a(x) = \max(k, R-k)/R$ and the soft target is $y(x) = k/R$. At training stage $t$ the loss is computed only over the admitted set
$$A_t = \{x : a(x) \ge \tau_t\},$$
where the threshold $\tau_t$ steps down from 1 toward $\max(\lceil R/2 \rceil, R - \lceil R/2 \rceil)/R$, the lowest agreement a pixel can have. The threshold moves to the next agreement level once the expected calibration error of the model on $A_t$, measured against $y$ on a held-out split, stops improving. Because $R$ is small (3 to 7 raters in the benchmarks below), the schedule has at most a handful of discrete stages. The one remaining setting is the calibration plateau tolerance, which is expressed in probability units rather than epochs. Our hypothesis is that this is what makes it portable.
Relation to Prior Work
Three existing methods motivate the proposal, and each supplies part of its mechanism. Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding anneals label smoothing from soft to hard, applies more smoothing at boundaries through spatially varying smoothing, and paces samples using a baseline network's confidence. It reports more than a 3% Dice gain over static smoothing on robotic instrument segmentation. Its boundary softness is synthetic, its endpoint is one-hot, and its pacing is precomputed. We replace all three with measured rater agreement, a soft endpoint, and online calibration.
Refining Context-Entangled Content Segmentation via Curriculum Selection and Anti-Curriculum Promotion down-weights high-entropy pixels early and switches phases at a fixed epoch $T_c$. Its authors state that $T_c$ must be retuned per domain and that an automatic transition remains open. We treat that admission as the target of our study, and we swap model entropy for human disagreement.
Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling (MRNet) builds pixel-wise disagreement maps and uses them as attention, but it trains on everything from the first step. We use the same kind of map to order training instead of to weight features.
Two further papers constrain the design. Beyond scalar losses: calibrating segmentation models via gradient vector field surgery shows that Dice-style gradients drive predictions toward extremes. It replaces them with a gradient field that scales with $|y-p|$ and reports reductions in negative log-likelihood and calibration error by factors of 4 to 6 in some cases. This is our main competing explanation, since it may remove sensitivity without any curriculum. Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum automates pacing through learned logit temperatures, but it distorts confidence and needs post-hoc recalibration. That paper is why we avoid pacing mechanisms that touch logits and gate on calibration instead. It also warns us to check that our curriculum does not leave its own mark on the final probability maps.
Finally, Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle argues that some disagreement regions carry diagnostic signal, such as cup-to-disc geometry in glaucoma. Because our curriculum admits contested pixels later but never discards them, and ends on their soft rater frequencies, it should keep that signal. We check this on downstream diagnosis where labels exist.
Research Program
The work has three parts. The first part is a sensitivity landscape. On each dataset we train a fixed backbone, a 2D U-Net, or a 3D U-Net for the CT tasks, under several curriculum families. Each family is swept over a shared grid of pacing duration, initial admitted fraction, and pacing shape (linear, root, geometric), with three seeds per cell. The families are synthetic spatially varying smoothing annealed to one-hot, entropy-based down-weighting with a fixed phase switch, agreement-ordered admission with a fixed epoch schedule, and agreement-ordered admission with calibration gating. Every family is crossed with three losses: soft Dice, cross-entropy on soft targets, and gradient vector field surgery. This factorial design separates the target effect from the loss effect.
The second part is portability. For each family and loss we tune on one dataset, freeze the settings, and apply them unchanged to every other dataset. We call the gap to that dataset's best-in-grid result the transfer regret.
The third part is a calibration-drift audit, following the warning from Data Parameters. For each configuration we compare the final probability maps to rater frequencies across agreement levels. The question is whether pixels admitted late end up systematically over- or under-confident relative to pixels admitted early.
Baselines
The naive reference point is a U-Net trained on majority-vote masks with Dice loss and no curriculum. This is what most clinical pipelines do. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation and DiFF both report that consensus labels underperform rater-aware training, so this baseline anchors the floor.
The most dangerous baseline is the same U-Net trained with cross-entropy on soft rater frequencies for the full run, with no curriculum. It has exactly our endpoint and none of our schedule. It gets the same epochs, augmentation, and learning-rate search as the best curriculum cell. Our method beats it only if calibration-gated agreement pacing matches or exceeds it on Q-Dice or soft Dice, and also converges faster or more stably across seeds. If it merely ties, we will report that the curriculum is unnecessary once targets are soft.
The strongest published methods are the multi-decoder U-Net from Uncertainty Quantification in Medical Image Segmentation with Multi-decoder U-Net and MRNet. The multi-decoder U-Net reports 0.852 on QUBIQ brain tumor task 1, 0.871 on task 2, 0.870 on kidney, and 0.553 on prostate task 1. For MRNet the file reports only a 1.2% soft Dice gain for the optic cup on RIGA and 29 frames per second. We found no absolute RIGA soft Dice for MRNet on Lacuna, so we will rerun it with our preprocessing and splits. For Gleason 2019, Pionono reports an unweighted Cohen's kappa of 0.758. Deep ensembles are the QUBIQ challenge's strongest family, with top teams at average Dice between 0.80 and 0.85 according to QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge. We run a five-member ensemble as an accuracy ceiling at five times the cost.
Because agreement pacing changes only the training order, it can also be added to MRNet or to the multi-decoder U-Net. Beating these methods therefore means that each architecture trained with agreement pacing improves on the same architecture trained without it, under equal budget.
The key ablation is agreement-ordered admission on a fixed epoch schedule. It isolates whether calibration gating, rather than agreement ordering itself, is responsible for any flattening of sensitivity.
Why not just…?
Why not just train on soft rater labels and skip the curriculum? This is the real threat, and the proposal is built so that it can win. Nothing in the file shows a curriculum beating soft-label training on multi-rater data. The factorial design reports the soft-label baseline in every table. If it equals the best curriculum cell on every dataset, we will conclude that the known sensitivity of segmentation curricula mostly reflected misspecified hard targets and that the fix is to drop the curriculum. That outcome is still useful. A win for the curriculum would have to show up where Uncertainty Quantification in Medical Image Segmentation with Multi-decoder U-Net reports instability: the QUBIQ brain-growth task, which has few annotations and widely spread raters.
Why not just use gradient vector field surgery with no pacing? It already removes the push toward extreme probabilities, which we hypothesize drives much of the sensitivity. Crossing every curriculum family with this loss measures directly how much sensitivity it removes on its own. If loss choice explains most of the variance in the landscape, the paper's conclusion becomes "fix the loss, not the schedule."
Why not just use a deep ensemble? QUBIQ shows that ensembles are top performers, but they multiply training and inference cost, which the challenge report flags as prohibitive for 3D CT. We report ensembles as a ceiling and compare them with single models at matched parameter count. We do not claim to beat them on accuracy alone.
Why not just tune a linear pacing function on agreement? That is our fixed-schedule ablation. It will probably reach similar best-case accuracy. The question is whether its transfer regret is larger, and that is exactly what the portability experiment measures.
Proposed Experiments
The datasets are the QUBIQ prostate, brain growth, brain tumor, and kidney tasks (3 to 7 raters, evaluated with Q-Dice at thresholds 0.1 to 0.9), RIGA (6 raters, soft Dice for disc and cup), REFUGE (7 raters, with glaucoma labels for a downstream AUC check in the style of DiFF), and Gleason 2019 (6 pathologists, Cohen's kappa). The primary outcomes are the interquartile range of the main metric over the pacing grid, transfer regret, expected calibration error against rater frequencies broken down by agreement level, and accuracy against the baselines above.
The decisive check is a comparison across families. Calibration-gated agreement pacing should show a narrower spread across the pacing grid and lower transfer regret than synthetic smoothing or entropy curricula, under both Dice and soft-label cross-entropy. It should also at least match the soft-label no-curriculum baseline. A second check is the calibration-drift audit: late-admitted pixels should show no confidence bias relative to early-admitted pixels. A third check, on REFUGE, is that downstream glaucoma AUC does not drop relative to training on all pixels from the start, which would indicate that the curriculum discarded diagnostic disagreement. If agreement ordering flattens sensitivity only under Dice loss, sensitivity is a loss problem. If it flattens sensitivity under every loss, rater agreement is a better difficulty measure than the proxies current curricula invent.
References
- Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding
- Refining Context-Entangled Content Segmentation via Curriculum Selection and Anti-Curriculum Promotion
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Beyond scalar losses: calibrating segmentation models via gradient vector field surgery
- Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Uncertainty Quantification in Medical Image Segmentation with Multi-decoder U-Net
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge