Labeling a 3D CT or MRI scan means tracing an organ or lesion on dozens to hundreds of 2D slices. Neighbouring slices are highly redundant, so a radiologist could label a handful and let a model fill in the rest. The open question is which handful. We propose to measure the value of labeling a slice the way the interpretability literature measures the importance of an input feature: remove it, retrain, and see how much worse the result gets. We then make that measurement cheap enough to use while choosing slices for new scans.
The core idea fits in two sentences. First, train a "completion network", one that takes a scan plus labels on a few slices and predicts the full 3D mask, on randomly chosen subsets of labeled slices. Every "what if this slice were not labeled" question then becomes an in-distribution query rather than an artifact, which amortizes the retraining that faithful importance estimates normally require. Second, check these cheap values against warm-started retraining on a small seed set. Only then distill them into a label-free scorer that ranks slices in new scans for annotation.
Core Claim
At a fixed budget of annotated slices, choosing slices by their retraining-faithful marginal value produces more accurate completed 3D labels than equally spaced slices, per-slice uncertainty, or whole-scan active learning. Spending the same number of slices spread across more patients also beats labeling fewer patients densely. The claim depends on one specific piece: the value estimate must agree with retraining. If the simpler masking estimate agrees equally well, or if equally spaced slices match the result, the claim fails.
Why This Gap Is Real
The retraining literature supplies both the motivation and the method. A Benchmark for Interpretability Methods in Deep Neural Networks (ROAR) removes pixels, retrains ResNet-50 from scratch, and shows that removal without retraining mostly measures the model's confusion at out-of-distribution inputs. Even random masking causes large accuracy drops unless the model is retrained. Reliable and Scalable Variable Importance Estimation via Warm-start and Early Stopping shows a cheaper route. It starts the reduced model from the full model's weights, runs a bounded number of gradient steps, and roughly halves the error of the masking ("dropout") estimate relative to full retraining. That estimate grows more biased as features become more correlated. Lazy Estimation of Variable Importance for Large Neural Networks linearizes the network around its trained weights and replaces refitting with a ridge regression, reporting over 10x speedups.
All three treat "variables" as tabular columns or 2D pixels, and none has been run on 3D segmenters. Adjacent slices are the most extreme case of correlated features, so the masking bias Sun and Raskutti document should be at its worst on them. LazyVI's guarantees rest on wide-network theory whose coverage of 3D U-Nets is untested. For that reason we use warm-start early stopping as the reference, not linearization.
The medical side supplies the application and shows that current practice selects at the wrong granularity. Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning (ASSFT) picks whole scans using foreground-restricted uncertainty and feature divergence. Positional Contrastive Learning for Volumetric Medical Image Segmentation exploits the fact that slices at similar relative depth look alike across patients, but it never asks which slice is worth labeling. Semi-supervised methods such as Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation and DyCON decide how to use unlabeled voxels, not which labels to buy.
Two findings about volumetric context constrain the design. Deep convolutional networks for pancreas segmentation in CT imaging shows that independent 2D predictions need cross-slice smoothing, which raised Dice from 56% to 68%. Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation reports degradation specifically at the first and last slices. Both suggest that the value of a slice is uneven through depth, concentrated where anatomy begins, ends, or changes shape. That is exactly what equally spaced selection ignores.
Research Program
The unit of annotation is a slab of $s$ adjacent axial slices, with $s$ matched to slice thickness so that thick and thin acquisitions are comparable. The completion network $f_\theta$ is a 3D U-Net. It takes the image volume and a mask channel that is empty except on revealed slabs, and it is trained on fully labeled training scans. During training, each sample draws a random revealed set $R$ of $k$ slabs, with $k$ itself randomized, and the loss is computed on the whole volume. For a scan $x$ with revealed set $R$, the value of slab $j$ is
$$v_j(R) = \mathrm{err}_\theta(x, R) - \mathrm{err}_\theta(x, R \cup \{j\}),$$
where $\mathrm{err}_\theta$ is one minus the Dice score of the completed mask against the full ground truth. Because $f_\theta$ has seen many reduced sets, querying it with $R$ and with $R \cup \{j\}$ is an in-distribution comparison. This is our analogue of ROAR's retraining, amortized into a single training run.
The first deliverable is a validation that this amortized value tracks retraining. On a seed set of about ten scans per dataset, we compute reference values by warm-starting from a completion model trained with slab $j$ always withheld from the pool, following Sun and Raskutti's recipe with a bounded step budget. For roughly twenty slabs, we also compute full retraining from scratch, as a gold check on the warm-start reference itself. Agreement is measured by Spearman rank correlation and by top-$k$ overlap.
The second deliverable is a label-free scorer, since new scans have no labels to compute $\mathrm{err}$. A small network $g_\psi$ reads the image, the current completion, and the completion's own predictive uncertainty per slab. It regresses the amortized $v_j(R)$ values, which are computed on training scans across many $R$. At deployment, slabs are chosen greedily: pick the top-scoring slab, reveal its (simulated) label, re-run completion, and re-score.
The third deliverable asks what the chosen slices look like. We expect selection to be non-uniform along depth, favouring organ extremities and shape transitions and spending more slabs on anisotropic, thick-slice scans. We test this directly against the boundary-slice degradation reported by the A&D paper.
Minimax enters as a design constraint that we test rather than assume away. A tempting variant replaces random reveals with an adversary $q_\phi$ that chooses which slabs to reveal so as to maximize completion error:
$$\min_\theta \max_\phi \; \mathbb{E}_{R \sim q_\phi}\big[\mathrm{err}_\theta(x, R)\big].$$
At equilibrium, the adversary's rarely revealed slabs would be the informative ones, and the network would be robust to any labeling choice. Convergence of Gradient Methods on Bilinear Zero-Sum Games proves that simultaneous gradient descent-ascent spirals outward on bilinear games and that positive momentum destabilizes them. Alternating updates and negative momentum on one player fix this in the bilinear case. Last-Iterate Convergence Rates for Min-Max Optimization obtains last-iterate convergence through gradient-norm (Hamiltonian) penalties, but these need Hessian-vector products that are costly for a 3D U-Net. We therefore run the adversarial arm with alternating updates and negative adversary momentum only. We report its seed-to-seed variance and the oscillation in the entropy of its reveal distribution, as well as its accuracy. Our prediction is that the adversary improves worst-case completion under poor human slice choices but does not improve the ranking. If so, the stable, non-adversarial version is the method.
Baselines
The do-nothing reference is equally spaced slabs at the same budget, which is what annotators do by default. It is also the most dangerous competitor, because through-plane redundancy may make even spacing nearly optimal for smooth organs. It is run through the identical completion network, so the only difference is which slabs are revealed. Beating it means a statistically significant Dice gain, paired across test scans over three seeds, at budgets of 5%, 10%, and 20% of slabs. The gain should be largest on Pancreas-NIH and on multi-organ AMOS, where shape varies most. Random slab selection is reported alongside as the floor.
The strongest existing selection method is ASSFT's foreground-restricted segmentation difficulty, applied at the slab level so that background voxels do not dominate. We also run ASSFT at its native scan level with the slice count matched. On AMOS2022-CT, ASSFT reports 80.51 Dice when querying 5% of scans, against 73.31 for random scan selection (ASSFT). We will reproduce the scan-level number with our backbone and then compare at an equal number of annotated slices. Beating it means our spread-out slab labels yield higher completion Dice and a better downstream segmenter at equal slice cost.
The downstream reference on Left Atrium is UA-MT. It reports 88.88% Dice and 7.32 95HD, the 95th-percentile surface distance, using 20% of scans fully labeled (UA-MT). We train UA-MT on our completed labels at the same total slice budget. For slice-level sparse selection itself, no reported result was found on Lacuna, so all selection comparisons are run by us under one protocol.
Two ablations isolate the key ingredient. The first computes $v_j$ by masking slabs in a completion network trained only with fixed dense reveals, which is the post-hoc estimate ROAR and Sun and Raskutti warn against. The second is the adversarial arm described above. All arms share data splits, backbone, training steps, and a tuning budget of twelve configurations each.
Why not just…?
Why not just label equally spaced slices? This is the real threat. If value is smooth in depth, even spacing is near optimal and our method only adds cost. The seed-set retraining maps settle it directly: if they are flat along depth, we report that and stop. If they peak at organ ends and transitions, as boundary degradation in the A&D paper and Roth et al.'s need for 3D smoothing suggest, equally spaced slabs must miss those peaks at low budgets.
Why not just rank slices by the model's uncertainty? Uncertainty marks where the model is unsure, not where a label would help most. A slab can be uncertain yet cheaply inferable from its labeled neighbours. ASSFT shows that raw entropy is dominated by background. Our slab-level ASSFT baseline and the correlation of uncertainty with retraining value quantify the gap, and $g_\psi$ is allowed to use uncertainty as an input, so it can only fall back to it.
Why not just mask slices in a trained model and measure the drop? ROAR shows that masking without retraining conflates information loss with distribution shift. Sun and Raskutti show the bias grows with correlation, and adjacent slices are highly correlated. The masking ablation, scored against warm-start and full retraining, measures how large this bias is on 3D data.
Why not just use an adversarial selector, which gives robustness for free? Simultaneous training of such games provably diverges in the bilinear case, and the stabilizers that work cheaply are untested at this scale. We run the stabilized adversary as an arm and keep it only if it matches the ranking quality of the random-reveal version.
Proposed Experiments
We use Left Atrium MRI, Pancreas-NIH CT, AMOS2022-CT, and Synapse. Seed sets for retraining validation contain about ten scans each, and the remaining scans are split into completion-training and test sets. The first decisive check is rank agreement. The amortized $v_j$ should correlate with warm-start retraining values substantially better than the masking estimate does, and warm-start values should themselves agree with full retraining on the gold subset. If the amortized values correlate no better than masking, the central mechanism fails.
The second check is completion Dice and 95HD at matched slab budgets against equally spaced, random, slab-level ASSFT, and matched-count scan-level ASSFT. We also report the downstream UA-MT and DyCON segmenters trained on completed labels. The third check is compute, meaning wall-clock time for amortized scoring versus warm-start and full retraining, reported as a cost-versus-error curve in the style of Sun and Raskutti. The fourth check is the stability and value of the adversarial arm, measured as variance across five seeds and as worst-case completion Dice under the adversary's own worst reveal sets.
References
- A Benchmark for Interpretability Methods in Deep Neural Networks
- Reliable and Scalable Variable Importance Estimation via Warm-start and Early Stopping
- Lazy Estimation of Variable Importance for Large Neural Networks
- Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning
- Positional Contrastive Learning for Volumetric Medical Image Segmentation
- Uncertainty-Aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation
- DyCON: Dynamic Uncertainty-aware Consistency and Contrastive Learning for Semi-supervised Medical Image Segmentation
- Deep convolutional networks for pancreas segmentation in CT imaging
- Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation
- Convergence of Gradient Methods on Bilinear Zero-Sum Games
- Last-Iterate Convergence Rates for Min-Max Optimization: Convergence of Hamiltonian Gradient Descent and Consensus Optimization