Medical image segmentation models miss small things. A brain MRI with an embolic stroke can contain dozens of separate lesions, each under a millilitre. Together they fill far less than one percent of the scan. When a network is trained to label every voxel, the background voxels supply almost all of the training signal. The tiny lesions are then the first thing the model gives up on. We propose splitting the problem in two, borrowing an idea from database systems. First, a cheap model answers a coarse question: which small cubes of the scan might contain lesion at all? Second, a separate fine model draws exact lesion boundaries, and it is trained and run only inside those cubes. The coarse stage is tuned on held-out data so that it misses at most a chosen fraction of true lesions. Under that guarantee, the fine stage sees a far more balanced problem than the whole-scan model does.
The point is not mainly speed. Moving from voxels to cubes changes the imbalance itself. Take a scan with roughly a million voxels and 100 scattered tiny lesions. At the voxel level, background outnumbers lesion by about 10,000 to 1. At the level of 8×8×8 cubes, there are about 2,000 cubes and up to about 100 positive ones, so background outnumbers lesion by only about 20 to 1. A lesion-presence classifier at that scale can be trained with ordinary losses. This is exactly the regime where the ISLES 2022 dataset paper reports that strong models such as nnU-Net, SegResNet and Factorizer systematically miss punctiform lesions (ISLES 2022).
Core Claim
For multi-focal, extremely sparse targets, we claim that most of the remaining error is in detection (finding each lesion) rather than delineation (tracing its boundary). If so, a two-stage system should beat a single dense network and match a three-model ensemble on lesion-wise recall at a fraction of the inference cost. That system has two parts. The first is a coarse cell-presence model with a calibrated recall guarantee. The second is a fine delineation model trained only on the cells the first model selects, so it learns under the same data distribution it will face at test time.
What is new relative to existing cascades is twofold. The candidate set is many small cells, not one organ crop. And its size is set by a calibrated miss-rate target, not a fixed threshold.
Where the idea comes from, and how it differs
The structural idea comes from the database literature on physical data independence. Physical Data Independence, Constraints, and Optimization with Universal Plans answers a query in two steps. It first expands the query into a deliberately redundant "universal plan" that contains every useful access path. It then prunes that plan, with a guarantee that no valid plan is lost in the expansion. GBB similarly stores entities in spatial "buckets", so that reasoning modules only search the regions that could be relevant. We take one lesson from these systems: over-generate with a completeness guarantee, then spend effort only where it pays. We do not claim to build a query compiler for neural networks. The research file's broader framing of logical query languages over voxels seems to us neither necessary nor testable. Our version of the "completeness guarantee" is statistical: a threshold calibrated on held-out scans so that the fraction of lesions with no overlapping candidate cell stays below a target $\alpha$.
Several medical segmentation papers motivate or constrain this design. Christ et al. cascade a liver segmenter into a lesion segmenter on the cropped liver. They improve liver Dice on 3DIRCADb from 72.9% to 93.1%. But their single hard crop permanently removes lesions at the organ border. Our design differs in three ways: the candidate set is many cells, it is recall-calibrated, and it is padded with context. Attention U-Net softly down-weights background inside one network, raising pancreas Dice on CT-150 from 0.814 to 0.840. It still computes over every voxel and still sums gradients over all of them. CaraNet targets small objects with reverse attention and reports 74.7% mean Dice on ETIS. It also remains a single dense pass.
Two loss-level methods constrain the design and also supply ingredients we reuse. Gradient-Aware loss shows that random patches often contain no tail-class voxels, which makes Dice gradients vanish or fluctuate. It fixes this with reweighting and hard-voxel mining, lifting Synapse tail organs from 0% Dice. Boundary loss shows that region losses are unstable for tiny objects. It replaces them with a distance-weighted contour loss, moving ISLES Dice from 0.575 to 0.656. Boundary loss has a weakness that matters here: on a background-only patch, its distance map is large and positive everywhere, so it punishes any foreground prediction. Because our fine model sees only candidate cells, and those cells contain lesion at a much higher rate than random patches, this failure mode should become rare. We will measure whether that holds rather than assume it.
Finally, DeepISLES runs three full 3D models (nnU-Net, SegResNet, Factorizer) and combines them by majority vote. It is the accuracy bar we must reach with lower compute.
Research Program
The coarse stage is a lightweight 3D network that outputs one presence probability per cell. Its inputs are a downsampled image plus pooled multi-channel intensities (diffusion and FLAIR for stroke; CT for abdominal tail organs). It is trained with cell-level cross-entropy. Cell size $s$ is a studied variable, swept over 4, 8 and 16 voxels per side.
We calibrate the threshold on a held-out split. We choose the largest threshold $\tau$ such that the empirical lesion-wise miss rate satisfies
$$\hat{m}(\tau) = \frac{\#\{\text{lesions with no candidate cell above } \tau\}}{\#\{\text{lesions}\}} \le \alpha.$$
Here $\alpha$ is the user-chosen miss budget; we test 0.02, 0.05 and 0.10. Because lesions within one patient are correlated, we calibrate over patients, not lesions, and report how well the stated budget holds on test centres that were unseen during calibration. Shift across centres is the setting where this guarantee is most likely to break. The ISLES 2022 data come from three centres, so we can test this directly by holding out a whole centre.
The fine stage takes each candidate cell plus a fixed margin of surrounding context. It is a standard 3D U-Net trained on samples drawn the way the calibrated coarse model selects cells at test time. That includes its false positives, so the fine model also learns to reject hard negatives. It uses Dice plus boundary loss, with distance maps computed on the whole volume and then cropped, so distances stay correct inside each cell. Overlapping cell predictions are averaged.
We will analyze three things. The first is how the class ratio seen by the fine model changes with $s$ and $\alpha$. The second is how the share of gradient coming from lesion voxels changes, compared with uniform patches and with nnU-Net-style foreground oversampling. The third is where the remaining errors fall: missed lesions versus boundary errors, split by lesion volume bins (under 0.1 ml, 0.1–1 ml, over 1 ml).
Baselines
The naive reference point is a single nnU-Net trained with its default recipe, which already oversamples patches containing foreground. This is the SEALS member of DeepISLES (DeepISLES). It is the most important competitor because it represents "just sample more lesion patches", the cheapest fix a reviewer would propose. We rerun it ourselves on the same ISLES 2022 splits with the same GPU-hour budget as our two stages combined. We found no reported single-model nnU-Net result on the ISLES 2022 test split on Lacuna, so all comparisons on that split use our own reruns.
The strongest existing method is the DeepISLES three-model ensemble. On the external JHCSC cohort of 1,685 scans it reports median Dice 0.82 and lesion-wise F1 0.86, at 1.5 to 2 minutes per scan (DeepISLES). We use the released ensemble as-is and evaluate it on the same splits. Beating it means one of two outcomes. Either we match its Dice within 0.01 and exceed its lesion-wise F1 on lesions under 0.1 ml, using less than a third of its inference time. Or we match its F1 at that cost. If JHCSC access is unavailable to us, we compare only on ISLES 2022 held-out centres and say so.
Two loss-level competitors test whether our gain could come from training objectives alone. One is a single nnU-Net trained with GDL plus boundary loss (Boundary loss); the other uses Gradient-Aware Dice and cross-entropy (Gradient-Aware loss). On Synapse, the second method's reported 66.39% mean Dice (versus 48.49% for its CPS baseline) is the reference point for the tail-organ experiment. That result is semi-supervised, so we match its labelled-data regime there.
A hard-cascade baseline in the style of Christ et al. replaces our cells with one dilated bounding box around the predicted brain or organ region. This isolates the effect of multi-cell candidates.
Three ablations isolate our key ingredients. The first keeps the coarse stage but uses an uncalibrated 0.5 threshold. The second keeps calibrated cells but uses a fine model trained on uniform patches, which tests whether matching the training distribution to the test-time candidates matters. The third replaces the coarse model with random cells under a matched compute budget. Tuning budgets, augmentation and training epochs are equalized across all methods.
Why not just…?
Why not just oversample foreground patches, as nnU-Net already does? This is the real threat. Oversampling also rebalances what the network sees during training, and it costs nothing at inference. Where the two approaches differ is at test time. An oversampled dense model still predicts over every voxel of a mostly empty volume. The Gradient-Aware loss paper notes that training on positive-heavy patches distorts the background statistics the model expects and produces false positives in whole-volume inference. In contrast, our fine model is trained on exactly the distribution it is applied to. The direct comparison against nnU-Net with matched compute settles this. If oversampled nnU-Net matches us on lesion-wise recall under 0.1 ml, the cell decomposition adds nothing beyond speed, and we will report it that way.
Why not just use a cascade? A cascade is our method with one candidate region and no calibration. Christ et al. show both its benefit and its single-point failure, which is that border lesions are lost. Stroke showers are scattered across vascular territories, so a single crop would either cover most of the brain or miss foci. The hard-cascade baseline measures this directly.
Why not just ensemble? DeepISLES proves that ensembling works. But every member processes the same dense volume, and the ISLES 2022 analysis reports punctiform misses even among the top models (ISLES 2022). Our claim is narrower: we expect similar accuracy at much lower cost, with an explicit miss-rate budget. If the ensemble keeps a clear recall lead on small lesions, our method's value shrinks to latency alone.
Why not just reweight the loss? Boundary loss and Gradient-Aware loss change how errors are weighted, but not which voxels get computed or which voxels supply gradients. They also remain in our fine stage. The loss-only baselines show whether removing background cells adds anything beyond them.
Proposed Experiments
The primary arena is ISLES 2022: 250 training cases from three centres, with 0 to 126 lesions per patient. We use five-fold cross-validation and a leave-one-centre-out split, and evaluate on JHCSC where access permits. Metrics are voxel Dice, lesion-wise F1 and recall split by volume bin, absolute lesion-count error, and inference time per scan on one GPU. A secondary arena is Synapse multi-organ CT, focusing on esophagus and adrenal glands, where baselines collapse to 0% Dice (Gradient-Aware loss). It tests whether the idea transfers from scattered lesions to single small organs. We expect the benefit to be smaller there, because the presence question is easier.
Three outcomes would be decisive. First, the calibrated miss rate on unseen centres should stay within about twice the target $\alpha$; if it exceeds that badly, the guarantee is not useful under shift. Second, lesion-wise recall under 0.1 ml should exceed oversampled nnU-Net by a margin larger than the spread across folds. Third, the distribution-matching ablation should lose most of the gain. That would show the improvement comes from what the fine model is trained on, not merely from removing background at inference. If the first check holds but the second fails, the result is still a cheaper DeepISLES-level system with a stated miss budget. If both fail, the detection-dominance hypothesis is wrong for this data, and the error breakdown will show where the errors actually come from.
References
- ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset
- Physical Data Independence, Constraints, and Optimization with Universal Plans
- GBB: A Generic Blackboard Development System
- Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields.
- Attention gated networks: Learning to leverage salient regions in medical images
- CaraNet: context axial reverse attention network for segmentation of small medical objects
- Gradient-Aware for Class-Imbalanced Semi-supervised Medical Image Segmentation
- Boundary loss for highly unbalanced segmentation
- DeepISLES: a clinically validated ischemic stroke segmentation model from the ISLES'22 challenge