A brain MRI with a small stroke lesion can contain ten million voxels, of which perhaps a thousand are lesion. A model that labels everything healthy is 99.99% accurate and clinically useless. That is the accuracy paradox: symmetric metrics reward the majority class. Ranking metrics such as the area under the ROC curve (AUC), which measure how often a lesion voxel scores above a healthy one, are the standard remedy. They fail here too. At this prevalence, a false-positive rate of just 1% means about a hundred false lesion voxels for every true one, so nearly the whole ROC curve describes operating points no radiologist would accept. The part that matters is a sliver at false-positive rates near $10^{-4}$, and its width depends on how rare the lesion is.
We propose to train 3D segmentation networks to directly maximize the area under that sliver. The window is defined in clinical units: "at most $\kappa$ false voxels per true lesion voxel." The method uses instance-wise partial-AUC machinery that costs linear time per patch and keeps running statistics across patches. Patches that contain no lesion, which is the common case at batch size one, still produce a ranking gradient. The learned threshold variable inside the objective doubles as the deployment threshold at the chosen false-positive budget. Two sentences for a colleague: Partial AUC with a fixed window is the wrong target when the foreground is 0.01% of the image, because the window must shrink with prevalence. We make it shrink, make it trainable on single 3D patches without memory banks, and test whether ranking itself, rather than hard-negative mining, is what helps tiny lesions.
Core Claim
The claim has three parts. First, for voxel-level lesion segmentation the clinically meaningful region of the ROC curve is a false-positive-rate window $[0, \beta]$ with $\beta = \kappa\,\hat\pi$, where $\hat\pi$ is the estimated lesion prevalence. Optimizing full AUC, or partial AUC at conventional windows such as $\beta = 0.05$ or $0.3$, mostly spends model capacity on irrelevant operating points. Second, the squared-surrogate one-way partial AUC (OPAUC) objective splits into per-class means, a tail mean of the hardest negatives, and variance terms. The positive-class statistics can therefore be tracked across patches as running averages. This removes both the stored-embedding memory bank of AUCSeg and the per-batch positive-sample requirement of PAUCI. Third, if positives are averaged per lesion (per connected component) rather than per voxel, small punctiform lesions count as much as large infarcts in the ranking. This targets the failure mode that the WMH challenge and ISLES 2022 both document for top methods.
What would be new is the combination: a prevalence-scaled window, buffer-free streaming statistics at batch size one, and a learned threshold at a clinical false-positive budget, all evaluated on 3D lesion benchmarks. The literature in the file has pieces of this but never together.
Where This Sits in Prior Work
The method comes from the instance-wise ranking literature. SPAM showed that for squared-surrogate AUC, the optimal auxiliary variables are class-conditional means that can be updated in closed form from running averages, with no sample buffer. FSAUC gave a primal-dual stochastic scheme with fast rates. PAUCI and its unbiased successor UPAUCI extended this to partial AUC. They replace top-$k$ negative selection with a learned, softplus-smoothed threshold, solve it with accelerated gradient descent-ascent, and report OPAUC at $\beta = 0.3$ of 0.9587 on iNaturalist2021 versus 0.9283 for an AUC-margin baseline. We reuse their decomposition and threshold variable. All of these works treat each sample as an independent image with one label. None faces patches with zero positives, spatially coupled voxels, or windows four orders of magnitude narrower than the ones they evaluate.
The closest existing work is AUCSeg. It optimizes pixel-level AUC for long-tailed 2D natural-scene segmentation with a squared surrogate and a tail-class memory bank that pastes stored embeddings into batches. It reports +1.05 mIoU on ADE20K at a cost of iteration time rising from 0.76s to 1.62s. It optimizes the full ROC curve, was never run on 3D volumes or extreme voxel skew, and relies on stored features. We differ on all three points: a partial and prevalence-scaled window, 3D volumes at batch size one, and scalar running statistics instead of a feature bank. Bucketed ranking losses solve a related speed problem for detection-level average precision by sorting logits and grouping negatives into buckets. That still costs $O(N\log N)$ in the number of candidates, which is awkward for $10^7$ dense voxels. Our objective needs no sort.
Three medical-segmentation papers motivate and constrain the design.
- GA Loss diagnosed that small organs vanish from 3D patches, so Dice gives zero gradient there. Its fix is a probability-aware Dice term plus top-$k$ hard-pixel cross-entropy, which raised Synapse DSC from 48.49% to 66.39%. This is the strongest evidence that hard-negative mining alone already helps. It is therefore the competitor our key ablation must beat.
- Boundary loss showed that overlap losses are unstable on tiny targets. Adding a distance-map term to generalized Dice raised ISLES Dice from 0.575 to 0.656 and WMH Dice from 0.727 to 0.748. This makes it the strongest reported number on our benchmarks in the file.
- MCC loss invoked the accuracy paradox directly and trained at batch size one. That is the regime we target. It also shows that symmetric correlation losses cannot select an operating window.
On the evaluation side, Rajkomar et al. and PharmaNet supply the motivating evidence that high AUROC hides false-positive floods; PharmaNet's 97.7% ROC-AUC coexisted with an earlier state of the art at 1% normalized average precision. Bounded DCA supplies the principle we borrow. It shows that AUROC averages over label shifts in a way set by the model's own score distribution rather than by any clinically plausible prevalence, and argues for clipping evaluation to realistic prevalence ranges. Our window is the training-time analogue of that clipping, expressed as a false-positive budget rather than a cost ratio. That framing also sidesteps the cost-elicitation problem the Bounded DCA authors raise.
The Objective
Let $f(x)$ be the network's score for voxel $x$. Let $\mu_+$ and $\sigma_+^2$ be the mean and variance of lesion scores, with each lesion component weighted equally. Let $\mu_{-,\beta}$ and $\sigma_{-,\beta}^2$ be the mean and variance of the top-$\beta$ fraction of healthy-voxel scores. With the squared surrogate and target margin 1, the OPAUC loss is
$$ \mathcal{L} = \sigma_+^2 + \sigma_{-,\beta}^2 + \big(1 - \mu_+ + \mu_{-,\beta}\big)^2 . $$
This follows from expanding $\mathbb{E}\,(1 - f^+ + f^-)^2$ over independent positive and tail-negative draws. The tail mean has the variational form
$$ \mu_{-,\beta} = \min_{s}\Big\{ s + \tfrac{1}{\beta}\,\mathbb{E}_{-}\big[(f(x) - s)_+\big] \Big\}, $$
where $s$ is a scalar threshold. Its minimizer is the $(1-\beta)$ quantile of healthy-voxel scores. Since $\beta = \kappa\hat\pi$, $s$ is the score threshold at the target budget of $\kappa$ false voxels per lesion voxel. We optimize $s$ jointly with the network, using PAUCI's softplus smoothing of $(\cdot)_+$. Everything is linear in the number of voxels in the patch.
Running statistics are what make batch size one work. The lesion-side quantities $\mu_+$ and $\sigma_+^2$ are exponential moving averages, updated whenever a patch contains lesion. This follows the closed-form updates SPAM justifies for fixed model parameters, with the drift caveat that the network changes. In a lesion-free patch, the running $\mu_+$ enters the margin term as a constant. The patch's hardest healthy voxels are therefore still pushed below where lesions currently score, and $s$ keeps tracking the tail. That is the ranking counterpart of GADice's absent-class regularizer. The difference is that it pushes negatives below a moving positive reference, not toward an absolute zero. The prevalence $\hat\pi$ is estimated once from training labels. $\kappa$ is the single exposed knob, read as false voxels tolerated per true lesion voxel.
Research Program
The work has three stages.
Build and verify the loss. We implement the loss for a 3D U-Net and verify on small volumes that it agrees with brute-force pairwise OPAUC. We measure the gap introduced by streaming $\mu_+$ as a function of moving-average decay.
Analyze the rare-voxel regime. With $\beta \approx 10^{-4}$, a single patch holds only a few dozen tail negatives. The quantile estimate $s$ is therefore noisy. We will measure how its variance, and the resulting bias in $\mu_{-,\beta}$, depend on patch size and $\kappa$. We will also test whether a slowly updated $s$ (a two-timescale scheme) is needed for stability, given that UPAUCI reports sensitivity to dual-variable initialization.
Evaluate on clinically framed metrics. The evaluation stratifies by lesion size and reports results at fixed false-positive budgets, not only Dice.
Baselines
The naive reference point is a 3D U-Net trained with the common Dice plus cross-entropy combination, thresholded at 0.5. It anchors how much any imbalance-specific method buys. We also report it thresholded post hoc at the validation threshold that meets the same false-positive budget $\kappa$ as our method. This separates "a better operating point" from "a better model."
The strongest published competitor on these benchmarks is generalized Dice plus boundary loss, following Boundary loss for highly unbalanced segmentation. It reports Dice 0.656 on ISLES and 0.748 on WMH, against 0.575 and 0.727 for generalized Dice alone. Those figures come from the older ISLES release and a 2019 pipeline, so we rerun the method in our pipeline rather than quoting them as targets. On ISLES 2022, no reported result was found on Lacuna for any loss-based method; the dataset paper gives cohort statistics only. Beating this baseline means a significantly higher prevalence-scaled partial AUC and higher lesion-wise recall at a matched false-positive count per scan on both datasets, with Dice no worse.
The most direct mechanistic competitor is GA Loss, which combines top-$k$ hard-pixel cross-entropy with gradient-aware Dice. It handles empty patches with a heuristic rather than a ranking objective. We use its supervised variant on the same backbone. Beating it on small-lesion recall at fixed false positives is the result that would show ranking adds something beyond mining.
The two ranking competitors are AUCSeg's full-AUC loss and PAUCI at its published window of $\beta = 0.3$, both ported to 3D. AUCSeg keeps its memory bank, and PAUCI uses per-patch statistics only. These test whether the window and the streaming statistics matter, separately from "use a ranking loss." We also include MCC loss as the batch-size-one correlation baseline.
All methods share architecture, patch size, intensity preprocessing, augmentation, iteration count, and hardware. Each gets the same number of hyperparameter trials on a held-out validation split. We report mean and spread over three seeds, plus wall-clock time and peak memory per iteration.
The ablations isolate each ingredient:
- a fixed $\beta = 0.05$ versus prevalence-scaled $\beta$;
- running versus per-patch positive statistics;
- lesion-weighted versus voxel-weighted positives;
- the decisive one: replacing the coupled margin term with pointwise cross-entropy on exactly the same top-$\beta$ negatives and the same lesion-weighted positives.
Why not just…?
Why not just train with Dice plus cross-entropy and pick the threshold that meets the false-positive budget? This is the most serious alternative, because the accuracy-paradox literature mainly asks for better operating-point selection, and threshold tuning provides it. But a threshold moves along a fixed ROC curve, while our objective tries to raise the curve inside the window. The experiment settles it. We compare full ROC curves within $[0, \kappa\hat\pi]$ for both models, and lesion recall at identical false-positive counts per scan. If the post-hoc-thresholded baseline matches us, the claim fails, and we will say so.
Why not just mine hard negatives with top-$k$ cross-entropy or focal loss, as GA Loss does? With a tiny window, the tail term of our loss also selects the hardest negatives, so the two methods overlap. GA Loss's Synapse gains show that mining works. The difference is the coupling. We push hard negatives below the running lesion score rather than toward zero, and an empty patch still sees the lesion reference. The pointwise-replacement ablation tests exactly this. If it matches the full loss, the honest conclusion is that the prevalence-scaled window and lesion weighting were the useful part. The paper would then be reframed around those, and the ranking framing dropped.
Why not just use AUCSeg or PAUCI as published? AUCSeg optimizes the whole ROC curve, so most of its gradient acts on false-positive rates that, at 0.01% prevalence, correspond to thousands of false voxels per lesion. Its memory bank also doubled iteration time on 2D scenes, by its own report. PAUCI's windows were designed for image-level class imbalance, not voxel prevalence, and it assumes each batch mixes positives and negatives. Both are run as baselines, and the fixed-$\beta$ and per-patch ablations show which of these two departures matters.
Why not just add boundary loss, which already penalizes distant false positives? Boundary loss shapes geometry around lesions the model has already found. Its own authors need a generalized-Dice warm-up because empty predictions give no contour gradient. It has no notion of an operating window or false-positive budget. It is complementary rather than competing, so we also report our loss combined with boundary loss.
Proposed Experiments
The primary benchmarks are ISLES 2022 and WMH. ISLES 2022 has 400 multi-center cases, 250 for training and 150 for testing including an unseen center, with infarcts from 0 to 482 ml and up to 126 separate lesions per patient. WMH has 60 training and 110 test scans across five scanners. Synapse is a secondary multi-organ test for tail organs, using a one-versus-rest extension per class, because GA Loss reports 0% DSC baselines there.
The primary metric is voxel-level partial AUC over $[0, \kappa\hat\pi]$, normalized by window width, at $\kappa \in \{0.25, 1, 4\}$. Secondary metrics are:
- lesion-wise recall at 1, 3, and 10 false-positive components per scan, in a free-response ROC style;
- recall stratified by lesion volume, separating punctiform lesions;
- Dice and Hausdorff distance at the learned threshold $s$ for our method and at validation-tuned thresholds for baselines;
- results on the unseen ISLES 2022 center and per WMH scanner;
- seconds per iteration and peak memory, set against AUCSeg's reported 0.76s-to-1.62s overhead.
Four checks are decisive.
- Ranking versus mining. The full loss must beat the pointwise top-$\beta$ cross-entropy ablation on small-lesion recall at fixed false positives. Otherwise the ranking component is not the source of gains.
- Window choice. Prevalence-scaled $\beta$ must beat $\beta = 0.05$ inside the clinical window while losing little full AUC.
- Batch-size-one training. With running statistics, training at batch size one must match per-patch statistics at the largest batch that fits in memory, at no worse time per iteration.
- Learned threshold. On test scans, $s$ learned during training must land within a small factor of the target false-positive budget. That would let it replace post-hoc threshold search.
If checks 1 and 4 both fail, the proposal reduces to a hard-negative-mining recipe, and we would report it as such.
References
- AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation
- Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm
- Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge
- ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset
- Stochastic Proximal Algorithms for AUC Maximization
- Fast Stochastic AUC Maximization with O(1/n)-Convergence Rate
- Closing the Approximation Gap of Partial AUC Optimization: A Tale of Two Formulations
- Bucketed Ranking-Based Losses for Efficient Training of Object Detectors
- Gradient-Aware for Class-Imbalanced Semi-supervised Medical Image Segmentation
- Boundary loss for highly unbalanced segmentation
- Semantic Segmentation of 3D Medical Images with 3D Convolutional Neural Networks
- Reply: metrics to assess machine learning models
- PharmaNet: Pharmaceutical discovery with deep recurrent neural networks
- Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs