Medical segmentation models need pixel-accurate masks that only radiologists or pathologists can draw. The usual workaround is to collect cheap labels, such as scribbles, single clicks, tumor diameter lines, or boxes drawn by students, and then ask an expert to fix only the worst of them. Everything depends on which labels the expert is shown. Current systems choose them by looking at a single trained model, or a pair of models, and flagging the places where prediction and label disagree. A snapshot of this kind cannot tell a wrong label apart from a correct label on a rare, hard structure, because both have high loss at the end of training. Expert hours then go to reconfirming correct annotations of difficult cases that the model simply has not learned yet.
Work on noisy-label classification shows that the history of a sample's loss across training separates these two cases. A hard but correct sample's loss rises early and then falls once the network learns it. A mislabeled sample's loss stays high and flat. We propose to record that loss history for each annotation unit, meaning each cheap label's footprint inside an image (a scribble-derived region on a slice, a box, or a mask derived from a diameter line). From these histories we rank which units a simulated expert should correct under a fixed review budget. The claim we can test is that ranking by loss history yields more segmentation accuracy per expert query than ranking by static disagreement or uncertainty, and that the gain comes mostly from not wasting queries on hard-but-correct units.
Core Claim and Position Relative to Prior Work
Three papers motivate the mechanism, and all of them stop short of this setting. Delving into Sample Loss Curve to Embrace Noisy and Imbalanced Data documents the two curve shapes (rise-then-fall for clean rare samples, persistently high for noise) and learns sample weights from whole curves. It works only in classification, needs a clean validation split for its meta-learning, and names dense prediction as future work. Learning From Noisy Labels via Dynamic Loss Thresholding fits a two-component Gaussian mixture to the gap between early and late loss in order to estimate the noise rate without clean data. Its estimate is 59.1% against 60.0% true noise on CIFAR-10. It too is image-level classification. Refining Context-Entangled Content Segmentation via Curriculum Selection and Anti-Curriculum Promotion (CurriSeg) brings mean and variance of loss over epochs into segmentation, including polyps. It does this for whole images under full-mask supervision and uses the result to reweight training, not to spend an annotation budget. Its pixel-level signal is static entropy.
The method we reuse for routing, and the main competitor, is Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images. It sorts crowdsourced boxes into three groups by comparing labels against predictions from a model trained on clean seed data and a model trained on consensus data: errors fixed automatically, errors fixed by a rule, and cases sent to a pathologist. Its discrepancies come from single checkpoints.
Our proposal keeps their routing idea and replaces the ranking signal with per-unit loss histories. It moves the unit of analysis from the whole image to the annotation inside the image, and it requires no clean seed set. On the weak-supervision side, Learning to Segment Medical Images with Scribble-Supervision Alone and CycleMix constrain the design. They show that expanding scribbles into full masks works well on ACDC, the cardiac MRI benchmark. Their checks on the expansion, which are Monte Carlo dropout (repeated forward passes with random units switched off) and connected-component shape priors, are static. CycleMix's shape priors cannot tell when a scribble was placed in the wrong tissue.
Research Program
The unit of analysis is the annotation unit $u$: the set of pixels that a cheap label, or the pseudo-mask expanded from it, assigns to one class on one slice or image. During a short probe training run we record the unit loss at each epoch $t$:
$$\ell_u(t) = \frac{1}{|u|}\sum_{p \in u} \ell(p, t),$$
where $\ell(p,t)$ is the per-pixel cross-entropy against the cheap or pseudo label at epoch $t$, and $|u|$ is the number of pixels in the unit.
Averaging within a unit addresses the variance problem that makes pure pixel-level curves noisy. It also keeps storage at one float per unit per epoch rather than one per pixel. The probe follows the cyclical learning-rate protocol of the loss-curve paper above, so that curves pass through repeated phases of learning and forgetting.
From each curve we extract features that need no clean data: early-minus-late loss, as in dynamic loss thresholding; mean and variance, as in CurriSeg; and the timing and height of the peak. We cluster these features into a "persistently high" group and a "rise-then-fall" group, using a two-component mixture or two-way clustering. This mirrors the unsupervised clustering of per-epoch loss vectors in Efficient Bias Mitigation Without Privileged Information. The ranking score is the posterior probability of the persistent group. The expert reviews the top $k$ units. The final model is retrained on corrected labels, and the remaining units are kept, down-weighted, or dropped according to their score.
To control cost on volumetric data, we test whether curves from a lightweight 2D proxy network rank units as well as curves from the full model. This borrows the proxy-trajectory idea that SmallToLarge (S2L) validated for language-model data selection.
Baselines
The naive reference points are random selection of $k$ units for expert review, and no review at all. The no-review case means training on the cheap labels with the same backbone and schedule. CycleMix reports exactly this setup at 76.6% Dice on ACDC for a plain UNet on scribbles.
The strongest existing method for deciding what the expert sees is the static two-model discrepancy ranking of Active Label Cleaning. On their private electron-microscopy data it reaches 67.18% average precision at 50% overlap (AP50), which is 95.79% of full-expert performance at 73.30% lower cost. On the public BCCD blood-cell set we found no reported result on Lacuna, so we will rerun it. To keep the comparison fair, we give it the same expert-labeled seed set it requires. We also run a version without the seed, in which the seed model is replaced by a model trained on cheap labels, because our method uses no seed. Both methods get the same backbone, epochs, and hyperparameter search budget.
The second static competitor ranks units by Monte Carlo dropout uncertainty at the final checkpoint. This is the signal used by Can et al., whose scribble pipeline reports 88.0% Dice on ACDC against 91.2% with full masks.
The third competitor ranks units by final-epoch loss alone, which isolates the value of having any history.
Two ablations target the key ingredient. One replaces the full-curve features with only mean and variance across epochs, which is CurriSeg's statistic applied per unit. The other computes curves per image instead of per unit, as CurriSeg does.
Beating a baseline means a higher Dice-versus-budget curve at matched expert budgets of 2%, 5%, and 10% of units, averaged over five seeds, with non-overlapping confidence intervals. It must also show a lower fraction of queries spent on units that were actually correct.
Why not just…?
Why not just use Monte Carlo dropout or prediction entropy, which costs one extra pass? Uncertainty is high both on wrong labels and on hard correct structures, such as the small apical and basal slices in ACDC. That ambiguity is exactly what the loss-curve paper and CurriSeg found had to be resolved with history. Our wasted-query metric measures this directly. If dropout spends no more queries on correct hard units than curves do, the extra cost of curves is not justified, and we will say so.
Why not just rely on CRFs or shape priors to clean pseudo-masks? Weakly Supervised Lesion Co-Segmentation on Ct Scans found that a dense CRF raised precision but cut recall on lesion boundaries. CycleMix notes that shape priors cannot detect a scribble placed in the wrong structure. Our corruption protocol injects exactly such misplaced scribbles, and we run CycleMix's regularizers alongside every ranking method. That way any gain is measured on top of structural cleaning, not in place of it.
Why not just use mean and variance of loss, as CurriSeg does, instead of curve shape? This is a real threat, because a two-number summary may capture most of the signal at lower cost. The mean-and-variance ablation settles it. If it matches full-curve features, we adopt it as the method. The remaining contributions are then per-unit granularity and budgeted expert routing.
Why not just drop high-loss units instead of paying an expert? Dropping removes coverage of rare anatomy. Our budget curves include an "auto-drop, no expert" arm at the same thresholds, which tests whether correction by the expert adds value over filtering.
Proposed Experiments and Decisive Checks
The main benchmark is ACDC scribbles, using the scribble annotations from the scribble-supervision literature. We corrupt a controlled fraction (10% to 40%) of units in two ways: swapping the class of a scribble, for example right ventricle to left ventricle, and relocating it into an adjacent structure. Clean small apical and basal slices serve as natural hard-but-correct units.
We repeat the study on MSCMRseg, the cardiac late-enhancement MRI set where CycleMix reports a gap from 37.8% to 80.0% Dice, which gives room for label errors to matter. On NIH DeepLesion we use masks expanded from RECIST diameter lines, and the 1,000 manual masks used by the co-segmentation paper serve as the expert oracle. On BCCD we inject the noise types defined by Active Label Cleaning (background, missed, mislocated, box-in-box) so that our results can be compared with theirs. The expert is simulated by revealing the ground-truth label of each queried unit.
The decisive check is precision at $k$ for detecting corrupted units, split into two strata: hard-correct units and easy-correct units. The method only earns its premise if it lowers false alarms on the hard stratum relative to static discrepancy and dropout. Secondary checks have three parts. First, robustness of the ranking when the probe uses SGD versus Adam and different batch sizes, measured by rank correlation of scores, which addresses the known sensitivity of trajectories to optimizer settings. Second, agreement between 2D-proxy rankings and full-model rankings. Third, wall-clock and memory overhead relative to training once without a probe.
References
- Delving into Sample Loss Curve to Embrace Noisy and Imbalanced Data
- Learning from Noisy Labels via Dynamic Loss Thresholding
- Refining Context-Entangled Content Segmentation via Curriculum Selection and Anti-Curriculum Promotion
- Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images
- Learning to Segment Medical Images with Scribble-Supervision Alone
- CycleMix: A Holistic Strategy for Medical Image Segmentation from Scribble Supervision
- Efficient Bias Mitigation Without Privileged Information
- SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
- Weakly Supervised Lesion Co-Segmentation on Ct Scans