In clinical segmentation, experts rarely trace lesions from a blank canvas anymore. They are shown a model's proposed outline and asked to fix it. That changes what an annotation costs. The time a radiologist spends on a scan depends on how many separate mistakes the proposal contains, how long their boundaries are, and how hard those mistakes are to spot. It does not depend on how big or complex the lesion is. Active learning, meaning methods that choose which unlabeled examples to send to a human, still mostly prices examples either uniformly or by the geometry of the predicted object. Both prices are wrong once the expert's job becomes correction. Worse, the cost and the value of an example are coupled. The scans the model is most confused about are exactly the scans whose proposals are worst, so the cheap verify-and-click workflow quietly turns back into expensive manual tracing on the most informative cases.
This note proposes to model annotation cost as the expected time to correct a proposal. It also treats the interface itself as a choice made per example: either the expert corrects the proposal, or the proposal is discarded and the expert draws scribbles from scratch. Selection then maximizes information gained per expected minute of whichever mode is cheaper. The saliency-noise literature supplies a specific, testable ingredient. A saliency map is a gradient-based picture of which pixels drove the model's output. Once its high-frequency noise is filtered out, it can show whether the model's proposal rests on the lesion or on surrounding context. Context-driven proposals look plausible but are wrong, and they are the errors experts must hunt for. We test whether that signal predicts correction time beyond what mask geometry and ensemble disagreement already capture.
Core Claim
Under a correct-the-proposal interface, per-example cost is a property of the proposal's errors, not of the target object. Errors are concentrated where uncertainty is high, so uncertainty-per-cost ratios computed with object-geometry costs are biased toward the cases where verification fails. A cost model built on proposal error structure, combined with a per-example choice between correcting and drawing from scratch, should give more segmentation quality per expert hour than either uniform-cost or geometry-cost selection. Denoised saliency is hypothesized to add predictive power for "hidden" errors. That last part is the riskiest piece of the claim, and the experiments are designed to falsify it cleanly.
What prior work establishes and leaves open
Three lines of work motivate the proposal. First, there is a long record that human labeling time varies with content. Settles et al. showed that uncertainty sampling drifts toward long, expensive sequences in text labeling. BAOD: Budget-Aware Object Detection tied annotation effort to object density in images. Bilmes et al. showed that greedy value-per-cost selection is near-optimal. That result supplies our selection rule, but it assumes the cost is known and independent of the value term. We break exactly that independence.
Second, two clinical studies drop the uniform-cost assumption. Cost-Sensitive Active Learning for Intracranial Hemorrhage Detection predicts labeling time linearly from the predicted hemorrhage's boundary length and number of connected pieces. It then solves a 0-1 knapsack under a radiologist time budget, picking the most uncertain set of scans that fits. This is the closest existing method. Its cost describes drawing the lesion, not fixing a proposal, and it never asks whether the proposal is shown to the annotator. nnActive standardizes 3D medical active learning on nnU-Net and introduces Foreground Aware Random sampling: random selection constrained so that queried patches contain anatomy. It finds that this simple baseline erodes most of the apparent advantage of sophisticated query methods. It also reports a Predictive Entropy Paradox: entropy sampling is good per sample but worst per annotated foreground voxel, because it targets complex boundaries. This constrains us. Any gain must survive comparison to Foreground Aware Random, and "foreground voxels" is itself only a proxy we intend to replace with measured time.
Third, the verification-versus-generation asymmetry noted by Schubert et al. and Goel et al. is what makes correction cost the right object. The from-scratch fallback is grounded in Learning to Segment Medical Images with Scribble-Supervision Alone. That work expands sparse scribbles through random walks, trains a segmentation network, and refines the result with conditional random fields and uncertainty filtering. It reaches 88.0% Dice on ACDC against 91.2% with full supervision. Scribbles are therefore a credible cheap mode, and we reuse that pipeline unchanged.
The saliency side supplies method and also a warning. Agreement Between Saliency Maps and Human-Labeled Regions of Interest found that dermatology models often attend to hair, lips, or nails rather than the lesion. Its analysis was post hoc and fed nothing back into labeling. Saliency strikes back traces gradient noise to pooling and strided convolutions and removes it with a tuned low-pass filter, FORGrad. It runs orders of magnitude faster than perturbation methods (about 0.78 s versus about 626 s per 100 images for RISE), which makes per-candidate attribution affordable inside a selection loop. AdaptGrad fixes SmoothGrad's out-of-range sampling at almost no added runtime. The warning is that FORGrad's authors also report that common faithfulness metrics reward high-frequency noise. We therefore never judge saliency by faithfulness scores here, only by whether it predicts measured human correction time.
Research Program
The work has three components.
The first is a timing study. Trained annotators, with clinicians on a subset, correct model proposals and also draw scribbles on the same cases, with every editing action logged. The cases are drawn from ACDC and KiTS2021 and stratified by model quality. This yields ground-truth correction and scribble times per case. No result on Lacuna reports such paired timings.
The second is a correction-cost predictor that sees only what is available before querying. Its inputs are features of the proposal and its predicted error structure: ensemble-disagreement regions, their count and boundary length, and the fragmentation of the proposal. Raw versus denoised saliency features are an explicitly ablated input. The main one is the fraction of attribution mass that falls outside the proposed region, computed after FORGrad or AdaptGrad cleaning. A parallel predictor estimates scribble time from simple target size and shape.
The third is selection. Let $U(x)$ be the information value of example $x$ (ensemble disagreement, as in the hemorrhage work), and let $\hat{C}_{\text{fix}}(x)$ and $\hat{C}_{\text{draw}}(x)$ be the predicted correction and scribble times. Each cycle chooses a set $S$ and a mode per example to solve
$$\max_{S} \sum_{x \in S} U(x) \quad \text{subject to} \quad \sum_{x \in S} \min\big(\hat{C}_{\text{fix}}(x), \hat{C}_{\text{draw}}(x)\big) \le B,$$
where $B$ is the expert-time budget per cycle. Because minutes are discretized, this is the same knapsack the hemorrhage work solves with dynamic programming. The only changes are the cost being minimized and the added mode decision.
For large-scale active learning runs, the fitted cost models charge simulated annotators. To avoid circularity, final results are reported in time charged by a separate cost model fit on held-out annotators. They are also replayed on the subset of cases with real logged times.
Baselines
The naive reference is uniform random selection with every example corrected regardless of proposal quality, charged at measured or modeled correction time. It shows how much of any gain comes from selection at all.
The strongest simple competitor is Foreground Aware Random sampling from nnActive. That paper shows it closes most of the gap to sophisticated query methods, which makes it the baseline a skeptic will reach for first. We run it inside the same nnU-Net configuration, with the same seeds, budget, and cycles. No reported numerical result was found on Lacuna for Foreground Aware Random on ACDC or KiTS2021, so we reproduce it.
The strongest existing cost-aware method is the knapsack selection of Cost-Sensitive Active Learning for Intracranial Hemorrhage Detection. It uses committee disagreement with the boundary-length and component-count cost of the predicted mask. We use the same ensemble size and uncertainty measure as our method, so the cost model is the only difference. That paper reports matching full-data performance with half the data, but no numerical Dice for that comparison was found on Lacuna. Uncertainty-only entropy sampling is included because the Predictive Entropy Paradox predicts it will waste time on proposals that are expensive to correct.
The scribble-only passive pipeline of Can et al. anchors the from-scratch mode. Its reported 88.0% Dice on ACDC is the reference for what scribbles alone achieve without selection.
Two ablations isolate the proposal's ingredients. One removes the mode choice and forces correction on everything. The other removes saliency features, or keeps them in raw, undenoised form.
Beating the field means a higher area under the curve of Dice against expert minutes than both Foreground Aware Random and the geometry-cost knapsack, across seeds, on both datasets. The win must hold when charged by the held-out cost model and on the real-time replay.
Why not just…?
Why not just smooth the proposals with morphological opening or a conditional random field and then reuse the hemorrhage paper's geometry cost? This is a real threat to one part of the claim. Cleaning proposals before showing them may recover much of what saliency denoising would. We therefore include a "cleaned proposal" arm, presented to annotators and used to compute geometry costs, and we do not claim that any particular denoiser is necessary. What smoothing cannot fix is the target of the cost. A smooth but misplaced proposal has short boundaries and few components, so the geometry cost calls it cheap even though correcting it means redrawing everything. The timing study measures exactly this gap.
Why not just use Foreground Aware Random and skip cost modeling? nnActive gives this real support. Foreground Aware Random beats nothing only through sensible screening, yet it rivals complex methods. If our method fails to beat it in expert minutes, the honest conclusion is that correction-cost modeling is not worth its complexity. We will report that outcome as such.
Why not just use ensemble disagreement as the cost signal, since errors live where models disagree? Disagreement may indeed explain most of the variance in correction time. If so, the saliency features are redundant, and the ablation will show no gain. The reason to expect otherwise is the shortcut finding of Singh et al.: models can be confidently and consistently wrong because they rely on context. Such errors produce low disagreement but take experts a long time to notice. We test this directly by measuring how much of the remaining variance in correction time, after disagreement is accounted for, the denoised saliency features explain.
Proposed Experiments and Decisive Checks
The first decisive check tests the cost model itself. On the timing data, does predicted correction time rank cases by measured time (Spearman correlation on held-out annotators) better than the hemorrhage paper's geometry cost and better than foreground voxel count? If not, the core claim fails before any active learning is run.
The second check tests the coupling claim. We measure the correlation between ensemble uncertainty and correction time, and the fraction of high-uncertainty cases for which scribbling was actually faster than correcting. A near-zero correlation would mean the uniform and geometry baselines are not biased in the way we claim.
The third check runs the full active learning comparison on ACDC and KiTS2021, with MSD Hippocampus as a small-structure stress test. The metric is Dice against cumulative expert minutes, with mode-choice and saliency ablations included.
The fourth check targets the saliency hypothesis. It compares raw and FORGrad- or AdaptGrad-denoised attribution features as cost predictors, using both a convolutional and a transformer backbone. FORGrad's authors report that transformers produce cleaner gradients, which predicts that denoising matters less for the transformer. This gives the saliency ingredient a falsifiable, architecture-dependent signature rather than a blanket claim.
References
- An Analysis of Active Learning Strategies for Sequence Labeling Tasks
- BAOD: Budget-Aware Object Detection
- Average-Case Active Learning with Costs
- Cost-Sensitive Active Learning for Intracranial Hemorrhage Detection
- nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation
- Deep Active Learning with Noisy Oracle in Object Detection
- LLMs Accelerate Annotation for Medical Information Extraction
- Learning to Segment Medical Images with Scribble-Supervision Alone
- Agreement Between Saliency Maps and Human-Labeled Regions of Interest: Applications to Skin Disease Classification
- Saliency strikes back: How filtering out high frequencies improves white-box explanations
- AdaptGrad: Adaptive Sampling to Reduce Noise