Buying Slices by the Click: Order-Aware Active Learning for 3D Medical Segmentation

When an expert annotates a CT or MRI volume, a slice is never drawn in isolation. Once a radiologist has contoured slice 40, the segmentation model can propose a contour for slice 41, and the expert only has to correct it. That correction takes a few clicks instead of a full tracing. Current active learning methods for medical segmentation still choose an unordered batch of slices and charge every slice the same unit cost. Active learning here means algorithms that pick which examples a human should label next. Because cost is flat, these methods cannot tell a cheap slice (next to one already annotated) from an expensive one (in a new patient, drawn from scratch). As a result they may be optimizing the wrong budget.

We propose to treat the expert's session as an ordered sequence. The cost of each slice depends on what was annotated before it: how close the nearest annotated slice in the same volume is, how good the model's proposed contour is at that moment, and whether the expert has to switch patients. The value of the sequence is the coverage and informativeness of the slices obtained. The method builds the sequence one slice at a time. At each step it adds the slice with the best ratio of marginal value to marginal cost, where cost is measured in simulated correction clicks, and it re-plans after every annotation. The claim to test is concrete. At a fixed click budget, this order-aware selection gives better segmenters than selecting the best unordered set and then sorting it by slice position. If that claim fails, sequence modeling adds nothing to this problem, and the experiments are designed to show that clearly either way.

Core Claim

The ordering matters only through one mechanism, and we want to say plainly which one. In a pool-based loop where the model is retrained after each batch, the order in which labels arrive does not change the final label set. Order matters only if it changes what the labels cost, or if the model is updated inside the session so that earlier annotations change later proposals. We build on both effects. Let $\sigma = (s_1,\dots,s_m)$ be the ordered list of slices annotated in a round, and let $S(\sigma)$ be its underlying set. We solve

$$\max_{\sigma}\; F(S(\sigma)) \quad \text{subject to} \quad \sum_{i=1}^{m} c(s_i \mid \sigma_{<i}) \le B,$$

where $F$ is a monotone submodular value function over slices, $c(s \mid \sigma_{<i})$ is the correction cost of slice $s$ given the prefix $\sigma_{<i}$ already annotated, and $B$ is the click budget for the round. $F$ is a facility-location mutual information score mixing uncertainty and coverage, as in USIM. Submodular means it has diminishing returns: adding a slice helps less when similar slices are already chosen.

What is new is the prefix-dependent cost. In Budgeted Sequence Submodular Maximization the costs are fixed per item and the value depends on order. Here the value is a set function and the cost depends on order. This is the structure an editing-based annotation workflow actually has.

Relation to Prior Work

Three medical active learning methods supply the setting and the strongest competitors, and all three share one assumption we drop.

Active Learning with the nnUNet and Sample Selection with Uncertainty-Aware Submodular Mutual Information Measure (USIM) selects unordered batches by maximizing facility-location mutual information over gradient embeddings. It reaches near-full performance with under 8% of Spleen and under 5% of Liver annotated. Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation selects slices by k-center coverage in a patient-aware contrastive embedding. It argues that slices are more cost-effective than whole volumes. Active learning for medical image segmentation with stochastic batches names the "redundancy trap," where uncertainty sampling keeps picking adjacent near-duplicate slices. It escapes the trap with random candidate batches. All three count cost in slices.

Our view reverses one part of the redundancy-trap story. Adjacent slices are redundant in value but also cheap in cost. The right answer is not to avoid neighbors outright but to price them correctly. The slice-versus-volume conclusion of the coreset paper was likewise reached under unit cost, and it may not hold under edit cost. We test that directly.

The sequence side supplies the method. Selecting Sequences of Items via Submodular Maximization models order through bonus edges on a directed acyclic graph and selects edges greedily (OMEGA). Budgeted Sequence Submodular Maximization adds non-uniform costs with a cost-effective greedy method (GBM) and a Pareto evolutionary method (POBM), evaluated on movies and courses. Near optimality of greedy strategies for string submodular functions with forward and backward curvature constraints bounds greedy performance on ordered strings through curvature conditions. We reuse the cost-effective greedy rule and the curvature style of analysis. We do not reuse the edge-graph formulation, because a slice's cost depends on its nearest annotated neighbor, not on a fixed set of pairwise bonuses.

Two papers constrain the design. Online Submodular Set Cover, Ranking, and Repeated Active Learning commits to a fixed ranking each round and names adaptive next-step selection as open. We work in that adaptive regime, but empirically, without claiming their guarantees. Budgeted stream-based active learning via adaptive submodular maximization needs slices to arrive in uniformly random order, which is false when slices are read along the z-axis. For that reason we stay pool-based and do not stream along the scan.

Interactive segmentation supplies the cost model. DeepEdit trains one network for automatic and click-refined 3D segmentation. It reports a Dice of 0.723 at 10 clicks on PROSTATEx lesions, which shows that click counts are a meaningful and simulatable currency. Sequential interactive image segmentation shows that carrying memory across images cuts clicks to reach 90% IoU on LeedsButterfly from 2.89 to 0.52. It also validates click reductions as time savings with 20 users. That work assumes the image order is given. We choose the order.

Research Program

Cost simulator. We build a click-cost simulator for 2D slice annotation on top of a DeepEdit-style interactive model. The cost of a slice is the number of simulated corrective clicks needed to reach a target Dice (0.9, and 0.85 for small structures). The clicks start from an initial proposal equal to the current segmenter's prediction, fused with the propagated mask of the nearest annotated slice in the same volume. We add a fixed volume-switch penalty, varied in a sensitivity sweep. We fit the relationship between simulated clicks and real time with a small timing study of two or three annotators on a few hundred slices. This checks the simulator before we trust it.

Selection algorithm. The value function $F$ is USIM's facility-location mutual information, so any gain comes from ordering and cost, not from a better value function. The algorithm is adaptive cost-effective greedy. At each step it predicts the cost of each candidate slice given the current prefix, adds the slice with the highest marginal value per predicted click, observes the realized clicks, and optionally applies a lightweight online update to the segmenter within the session, in the spirit of SIIS's online purification step. Predicted cost comes from a small regressor on the proposal's confidence and its distance to the nearest annotated slice. That regressor is trained on simulator runs from the labeled pool only.

Theory. Our analysis target is modest. We ask whether cost-effective greedy keeps a constant-factor guarantee when cost satisfies a "cost diminishing" condition. That condition says a slice's cost never rises as the prefix grows, which is analogous to the forward-curvature conditions of Zhang et al. We also ask how the bound degrades when the condition is violated, for example when online updates occasionally make proposals worse.

Baselines

The naive reference is uniform random slice selection under the same click budget: random slices are drawn and charged their simulated cost until the budget runs out. Gaillochet et al. note that random selection is hard to beat because it is naturally diverse.

The strongest existing selector is USIM (facility-location mutual information), run with the same nnU-Net 2D configuration, the same embeddings, and the same number of rounds. It is the best-documented submodular method on the MSD tasks. USIM reports near-optimal performance with under 8% of Spleen, under 5% of Liver, and about 30% of Hippocampus annotated, all measured as a fraction of data, not clicks. We re-run it under click accounting and give it the most favorable in-volume order, described next.

The ablation that isolates our key ingredient is select-then-order. USIM picks the set as if every slice cost one unit. The slices are then annotated in the best order we can find for that fixed set, grouped by volume and sweeping outward from the most confident slice, and charged simulated clicks. Beating select-then-order shows that choosing the set with prefix-dependent costs in mind matters, not merely annotating in a good order.

A second ablation, static cost-aware selection, runs budgeted greedy with each slice's predicted cost fixed at round start. This matches Budgeted Sequence Submodular Maximization's fixed-cost setting and separates the benefit of static cost awareness from adaptive re-planning.

We also run three further baselines under the same click accounting and the same total selection compute:

  • Stochastic Batches with entropy, where Gaillochet et al. report 3D Dice rising from 67.01% to 71.27% on PROMISE12 at unit cost.
  • The GCL k-center coreset, which reports up to a 10% gain on ACDC at a 2–5% budget with scribbles.
  • A whole-volume interactive annotation baseline that fully annotates randomly chosen volumes with propagation.

Every method gets the same hyperparameter search budget on a held-out split. Beating a baseline means higher final 3D Dice at equal clicks, averaged over five seeds with non-overlapping 95% intervals, or equal Dice with at least 20% fewer clicks.

Why not just…?

Why not just pick the best unordered set with USIM and sort it along the z-axis? This is the most serious threat. Once a set is fixed, ordering within a volume is nearly trivial. If joint selection only reorders the same slices, it brings nothing. The select-then-order ablation is built to settle this. Our prediction is that joint selection chooses different slices: neighbors of already-annotated slices that are cheap but moderately informative, in place of isolated slices that are expensive and highly informative. If the two methods pick sets whose Jaccard overlap exceeds roughly 0.8 and reach Dice within noise, we will report that ordering does not matter at the selection stage.

Why not just annotate whole volumes interactively with propagation? Under edit-based cost, a full volume may be cheap enough that slice-level active learning loses its advantage. The coreset paper's case for slices was made under unit cost, so it does not answer this question. The whole-volume baseline tests it directly, and a loss to that baseline would be a useful negative result for the field.

Why not just use uncertainty sampling, since adjacent picks are now cheap? Once adjacent slices are cheap, the redundancy trap might look like a free lunch. It is not, because redundant slices still add little value. PROMISE12's gap between entropy and Stochastic Batches (67.01% vs 71.27%) shows that value redundancy costs accuracy even when labels are free to count. Uncertainty sampling under click accounting shows whether cheapness offsets that loss.

Proposed Experiments

We run on MSD Spleen, Liver, and Hippocampus, matching USIM's tasks, together with PROMISE12 and ACDC. Each task has five rounds with click budgets matched to the low-budget regime in the papers (roughly 2–10% of the full-annotation click cost). Metrics are 3D Dice against total simulated clicks, Dice against real annotator minutes on the timed subset, and the area under the Dice-versus-click curve.

Three checks decide the outcome. The first is joint selection against select-then-order at equal clicks. The second is adaptive against static cost awareness. The third is a sweep of the volume-switch penalty, which tests whether gains grow as sequence effects become stronger, as the mechanism predicts. If gains stay flat as the penalty rises, the story is wrong.

For click-budgeted active learning on these benchmarks, we found no reported result on Lacuna. The click-accounted numbers for all baselines will therefore be produced here under a shared protocol. The published unit-cost figures cited above serve only as sanity checks that each re-implementation matches its original.