Radiologists already mark most lesions they measure. They draw a diameter line on the single slice where the lesion looks largest, and those bookmarks pile up by the tens of thousands in hospital archives. Turning one such mark into a full 3D outline is still hard. A promptable segmenter is a model that outlines an object once you draw a box around it; MedSAM is the medical version. It needs a box on every slice. The cheapest current fix draws boxes on a few slices and linearly interpolates the rest. For small lesions this fails twice. A lesion a few millimetres across can start and end between two annotated slices. And a box that is off by three pixels barely matters for a liver but destroys the prompt for a 12-pixel nodule.
We propose a small, slice-aware "box propagator". It takes one clinician box on a lesion's key slice plus the surrounding CT stack, and predicts, slice by slice, whether the lesion is present and where its box lies. Those boxes then go to a frozen MedSAM. The key design choice comes from the small-object detection literature. Overlap-based matching and losses collapse when objects are tiny, so the propagator is trained with a scale-normalized loss and judged by downstream segmentation quality, not by box overlap. If it works, existing clinical bookmarks become 3D small-lesion masks at the cost of one box per lesion.
Core Claim
Two failures that are usually treated separately are the same geometric failure. Detectors miss small objects because overlap-based anchor assignment collapses for them. The textbook case is a 12×12 target against a 64×64 anchor, which gives an overlap ratio (IoU) of about 0.035. Overlap-based regression losses also stall near zero overlap (DAFE-FD, Improved Faster R-CNN for Liquid Bag Foreign Body Detection).
Propagating a box across slices hits the same wall. A small lesion that shifts a few pixels between slices has near-zero overlap with its own box on the neighbouring slice. So both linear interpolation and overlap-trained propagation are poorly suited to small targets. Our claim has two parts:
- If the clinician's single box is treated as a fixed, correctly sized anchor in-plane, the only thing left to learn is the out-of-plane motion and extent.
- If that motion is learned by a 3D-context model with a loss that does not depend on overlap, then small-lesion Dice and lesion recall can exceed sparse interpolation even when interpolation is given three to ten times more human boxes.
Relation to Prior Work
Three works motivate the gap.
Efficient Semi-automatic Segmentation-Labeling of Any Volumetric Medical Image interpolates boxes between sparse annotated slices and prompts MedSAM. It reaches 0.831 Dice on the Medical Segmentation Decathlon at 33% of slices, and 0.769 at 9%. It needs at least two annotated slices per object and assumes boxes change linearly between them. Our method needs one slice and replaces the linear assumption with a learned, image-driven prediction that can also say "the lesion has ended."
A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation shows that cropping around a detected box before segmenting raises DeepLesion Dice from 34.1% (whole-image nnU-Net) to 62.6%. It is strictly 2D and names the move to 3D as its main open step. We take its box-first design but supply the third dimension from a human seed rather than a detector.
DeepLesion mined bookmarks into boxes but produced no voxel masks. We aim to convert exactly that kind of bookmark into 3D masks.
Two works supply the method. Deep Volumetric Universal Lesion Detection using Light-Weight Pseudo 3D Convolution inflates a 2D ImageNet network into axial, coronal and sagittal 2D kernels ("pseudo-3D"). This adds volumetric context cheaply and raised sensitivity for lesions under 2 cm from 22.4% to 30.3%. We reuse that backbone design for the propagator. That paper does detection from scratch, with no human seed and no dense masks. MedSAM (Segment anything in medical images) is the frozen segmenter that receives our boxes. Its reported reduction of more than 82% in expert time is the regime we push further.
One work constrains the design. Deep convolutional networks for pancreas segmentation in CT imaging showed that post-hoc 3D smoothing lifts 2D pancreas Dice from 56% to 68%. Smoothing is a low-pass filter, though, so we do not rely on it for tiny structures.
Research Program
Measuring prompt tolerance. We first measure how tolerant MedSAM is to box error as a function of lesion size. On ground-truth masks from the Decathlon's small-target tasks (lung, pancreas tumour, colon cancer, hepatic-vessel tumour) we perturb the true per-slice box by offsets and scalings. Error is expressed both in pixels and as a fraction of lesion size. If the tolerated error is roughly a constant fraction of size, the required absolute precision for small lesions is a handful of pixels. That would justify the size-normalized loss and tell us what precision the propagator must hit.
The propagator. The model is a pseudo-3D network in the style of Cai et al. Its input is a crop centred on the key-slice box, spanning ±K slices, with the box itself supplied as an extra input channel. For each slice offset $k$ it outputs a presence probability $p_k$ and a box with centre $(\hat x_k, \hat y_k)$ and size $(\hat w_k, \hat h_k)$. It is trained on slices where the lesion is present (true centre $(x_k, y_k)$, size $(w_k, h_k)$) with
$$ \mathcal{L}_k = \frac{(\hat x_k - x_k)^2}{w_k^2} + \frac{(\hat y_k - y_k)^2}{h_k^2} + \Big(\log\frac{\hat w_k}{w_k}\Big)^2 + \Big(\log\frac{\hat h_k}{h_k}\Big)^2, $$
plus a binary cross-entropy term on presence for every slice. Every term is scale-free, so a 3-pixel error on a 10-pixel lesion is penalized as heavily as a 30-pixel error on a 100-pixel organ. The gradient also never vanishes at zero overlap, which is the failure the overlap-based losses suffer.
Training data. Ground-truth 3D boxes come from masks in AMOS22 and from Decathlon tasks held out of each evaluation fold. Training seeds are simulated as the largest-area slice of each connected component, jittered to mimic human drawing error.
A segmentation-aware variant. A second variant fine-tunes the propagator through MedSAM's frozen decoder. The aim is that boxes are chosen for the masks they produce, not only for matching ground-truth boxes.
Anisotropic data. Thick-slice volumes are handled by giving the model the physical slice spacing as an input. The presence head therefore learns extent in millimetres rather than in slice counts.
Baselines
All methods receive the same CT volumes, the same frozen MedSAM checkpoint, and the same simulated key-slice boxes with identical jitter. Any learned component gets the same training sets and the same hyperparameter search budget on a validation split.
Naive reference (copy-box). The key-slice box is copied to neighbouring slices up to a fixed extent chosen on validation data. This is what a user gets with no learning.
Upper reference (full manual). MedSAM is prompted with ground-truth boxes on every slice where the lesion appears. Kordt et al. report 0.848 Dice for this setting on the Decathlon.
Strongest existing sparse method. Kordt et al.'s bilinear interpolation, which reports 0.831 Dice at 33% of slices and 0.769 at 9%. Because it needs at least two slices, we run it at 2, 3 and 5 annotated slices per lesion and compare effort-matched and effort-unmatched curves. Beating it means one of two things on lesions below the median size:
- higher Dice at one box per lesion than interpolation achieves at two or three; or
- equal Dice at a third of the boxes.
These gains must come with no loss on large lesions.
Native-3D competitor. SegVol (SegVol) is given a 3D box built from the key-slice box with a validation-tuned depth margin. On the Decathlon sparse regime we found no reported result on Lacuna for SegVol, so we run it ourselves. The file reports SAM-Med3D at 0.556 and SAM-Med3D-turbo at 0.661 average Dice in that regime, from Kordt et al.
Mask-chaining. MedSAM's own predicted mask on slice $z$ is converted to a padded box for slice $z+1$. Propagation stops when the predicted mask vanishes.
Ablations isolating the key ingredients. We swap our loss for an IoU-style box loss to test the scale-normalized loss. We swap the pseudo-3D backbone for a 2D one that sees only the target slice and the seed to test volumetric context.
Why not just…?
Why not just give SegVol a 3D box and let a native 3D model handle context? This is the most serious competitor. SegVol keeps full volumetric context and zooms into a region before segmenting. Its strong reported number (0.8593 on AMOS22) is on organs, not small lesions. The only native-3D numbers in the sparse lesion-heavy regime are SAM-Med3D's 0.556 and 0.661, which trail interpolated MedSAM. That is suggestive, but SegVol is a different model. Our size-stratified comparison settles it directly. If SegVol matches the propagator on small lesions, the contribution reduces to the size-normalized prompt-tolerance analysis, and we will report that.
Why not just chain MedSAM's own masks slice to slice? This needs no training and could plausibly work for lesions that change slowly. We expect two failures. Small errors compound over slices into drift. And MedSAM, given a box, tends to produce some mask even after the lesion ends, so extent gets overshot. Both show up in our metrics: Dice by distance from the seed slice, and extent error in millimetres. If chaining matches the propagator, we will say the learned model is unnecessary.
Why not just smooth slice-wise outputs in 3D? Roth et al. show smoothing helps pancreas-scale structures (56% to 68%). A low-pass filter erodes structures a few voxels wide. We include Gaussian-smoothed copy-box and chaining outputs as additional rows to measure this directly on small lesions.
Why not just run 3D nnU-Net and skip prompting? nnU-Net scores 0.16 Dice on colon cancer and 0.21 on pancreas tumours in the Decathlon (The Medical Segmentation Decathlon). It scored 34.1% on full DeepLesion slices in Cheng et al. Fully automatic segmentation of small lesions is not yet reliable enough to replace a one-box workflow, so we include nnU-Net only as a context row.
Proposed Experiments
Primary benchmark. The primary benchmark is the Decathlon's lesion tasks, using leave-task-out training so the propagator never sees the evaluated pathology. Every connected lesion component is binned by diameter. We report:
- per-lesion Dice;
- lesion recall, the fraction of lesions with Dice above 0.1;
- slice-extent error in millimetres;
- Dice against human effort (boxes per lesion), so our method can be placed on Kordt et al.'s effort curve.
Decisive check. The decisive check is the smallest size bin. The claim fails if, in that bin:
- one-box propagation does not beat two-slice interpolation; or
- the IoU-loss ablation performs as well as the scale-normalized loss; or
- the 2D ablation performs as well as the pseudo-3D one.
The prompt-tolerance study must also show that the tolerated error shrinks with size; otherwise the motivation for scale-free training is wrong.
Transfer to real bookmarks. We seed the propagator with boxes derived from DeepLesion's measurement bookmarks, following Yan et al.'s padding of the diameter line. There are no 3D ground-truth masks on Lacuna for this set. We therefore check that the predicted mask's long axis on the key slice agrees with the recorded diameter, and a radiologist rates extent plausibility on a sample. This tests whether archives can be converted to 3D masks with zero added clicks. It also lets us compare against Cheng et al.'s GrabCut-derived 2D masks, which reached 62.6% Dice under their own 2D protocol. Agreement with those masks is a sanity check, not a target, since they are themselves heuristic.
References
- DAFE-FD: Density Aware Feature Enrichment for Face Detection
- Improved Faster R-CNN Network for Liquid Bag Foreign Body Detection
- Efficient Semi-automatic Segmentation-Labeling of Any Volumetric Medical Image
- A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation
- DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning
- Deep Volumetric Universal Lesion Detection Using Light-Weight Pseudo 3D Convolution and Surface Point Regression
- Segment anything in medical images
- Deep convolutional networks for pancreas segmentation in CT imaging
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- The Medical Segmentation Decathlon