Radiologists already annotate tumors every day, but not in a form segmentation models can use directly. When they assess treatment response, they draw two perpendicular lines across a lesion on the one axial slice where it looks largest. This routine measurement is called RECIST (Response Evaluation Criteria in Solid Tumors). Hospital archives hold tens of thousands of these marks. Full 3D outlines, where an expert traces the lesion on every slice, are rare and expensive.
The proposal is to train 3D lesion segmenters on many RECIST-marked scans plus a small set (around 5–10%) of fully outlined volumes, and to use the fully outlined volumes for a specific purpose. Instead of only adding them to the training loss, we use them to learn a map of where pseudo-labels derived from RECIST marks are trustworthy. That trust depends on how far a voxel sits from the measured slice, how close it is to the guessed boundary, and how faint the local tissue contrast is. Training on the weak scans then counts each voxel in proportion to that learned trust.
The trick that makes this possible is simple. Every fully outlined volume can also be given a simulated RECIST mark. Running the weak-label pipeline on it and comparing the result against the true outline tells us exactly where weak-derived labels go wrong.
Core Claim
In weak-to-3D lesion segmentation, errors are not spread evenly across a lesion. They concentrate in two places: slices far from the measured one, and boundaries where the lesion fades into similar-looking tissue. We claim that the right unit for down-weighting unreliable weak supervision is therefore the voxel's position relative to the RECIST anchor, not the whole training example.
We further claim that a small set of 3D expert masks is enough to estimate this position-dependent reliability, because each expert mask yields a paired weak label for free. Concretely, we expect a model trained this way to keep accuracy roughly flat with distance from the RECIST slice, where purely weak pipelines degrade. We also expect it to sharpen low-contrast boundaries without the recall collapse seen when boundaries are snapped to image edges.
What Prior Work Leaves Open
The closest prior work turns RECIST marks into 3D masks with no expert volumes at all. Self-Paced 3D Mask Generation from RECIST seeds GrabCut, a classical graph-cut foreground extractor, from the RECIST lines on the measured slice. It then grows predictions outward slice by slice, reaching 93% Dice on RECIST slices but 76% on full 3D volumes in DeepLesion. On a lymph node set it reports 70.2% Dice against 71.0% for full supervision.
Slice-Propagated 3D Mask Generation from 2D RECIST formalizes this propagation and reports 91.5% 2D and 76.4% 3D Dice. Both papers motivate our work, because accuracy decays as propagation moves away from the anchor slice. Neither paper has any way to know where the decay happens, which is what a small expert set can supply.
Hybrid supervision methods supply the idea of letting strong labels referee weak ones, but they operate on the wrong unit or the wrong dimension.
- Label-Efficient Hybrid-Supervised Learning learns one weight per weak example by comparing its gradient to the gradient from strong examples. It reaches 82.56% Dice on polyps with 10% strong labels, versus 82.91% with full labels, but only in 2D. A single weight per lesion cannot express "the core is right, the far slices are wrong." - Constrained-CNN losses calibrates global size bounds on a strong subset. A scalar volume bound says nothing about where a boundary is. - Multi-Label Deep Supervision aligns coarse patch labels in 2D retinal scans (OCT, optical coherence tomography).
Boundary-focused methods constrain the design rather than supply it.
- PnPNet sharpens confused 3D interfaces by pushing apart and pulling together voxel features. Contour-loss methods add an edge branch, reducing 95% Hausdorff distance on spleen CT from 7.57 to 5.88. Both require dense ground truth everywhere they apply a boundary loss. - Weakly Supervised Lesion Co-Segmentation found that a dense CRF (conditional random field, a post-processing step that aligns boundaries with intensity edges) cut recall sharply on low-contrast CT lesions. Boundaries were snapped to the wrong edges.
The design consequence is that boundary terms must be applied only where pseudo-boundaries are known to be reliable. That is exactly what the learned reliability map provides.
Research Program
Weak-label constraints. A RECIST mark carries more geometry than its usual use as a GrabCut seed. The voxels along both axis segments are certainly lesion. The four endpoints lie on the lesion boundary in-plane. We turn these into hard constraints: segment voxels are foreground, and the endpoints should sit at the model's 0.5 probability level. The unknown head-to-foot extent is handled by an extent prior: the distribution of 3D extent divided by long-axis length, estimated from the expert volumes.
Pseudo-labels and reliability. An initial 3D pseudo-label comes from the propagation pipeline of the Cai et al. papers. For each expert volume, we simulate its RECIST mark from the true mask, run the identical pipeline, and record whether each voxel's pseudo-label matches the expert label.
A small reliability model $r(v)$ is fit to predict, for voxel $v$, the probability that its pseudo-label is correct. It sees four features:
- the distance in millimetres from the RECIST slice, so anisotropic slice spacing is handled physically;
- the signed distance to the pseudo-boundary;
- the local intensity contrast across that boundary;
- the slice thickness.
We use gradient-boosted trees, or a two-layer network if the trees saturate.
Training objective. On weak volumes the segmenter minimizes
$$\mathcal{L}_{\text{weak}} = \sum_{v} r(v)\,\ell_{\text{CE}}(p_v, \hat y_v) + \lambda \sum_{v \in B_\tau} r(v)\,\ell_{\text{bd}}(p_v, \hat y_v) + \mu\,\mathcal{L}_{\text{RECIST}},$$
where:
- $p_v$ is the model's foreground probability and $\hat y_v$ the pseudo-label;
- $\ell_{\text{CE}}$ is cross-entropy;
- $B_\tau$ is the set of voxels within the pseudo-boundary band whose reliability exceeds $\tau$;
- $\ell_{\text{bd}}$ is a boundary loss, either a signed-distance surface loss or the push/pull terms of PnPNet;
- $\mathcal{L}_{\text{RECIST}}$ enforces the segment and endpoint constraints exactly.
Expert volumes receive ordinary Dice plus cross-entropy and an unrestricted boundary loss.
Iteration. After each round, the current model regenerates the pseudo-labels and $r$ is re-fit. To avoid leakage, the re-fit uses cross-fitting: $r$ is calibrated on expert volumes held out from the segmenter's own strong loss in that round.
Architecture. The backbone is deliberately conventional: a 3D nnU-Net-style network with anisotropic kernels. We test whether linear-time cross-slice mixing in the style of TranSamba adds anything as a separate ablation, not as part of the core claim.
Baselines
All methods share the same data splits, backbone where applicable, number of training epochs, and a tuning budget of an equal number of hyperparameter trials. Trials are selected on the same held-out expert-annotated validation volumes.
Strong-only baseline. The naive reference trains the 3D backbone on the 5–10% expert volumes alone. This is the do-nothing option a clinic could deploy tomorrow. Prior evidence suggests it loses to hybrid training: Constrained-CNN reports that 5 full plus 145 weak patients outperformed 25 full patients. We nonetheless rerun it at every strong fraction.
Purely weak propagation. We reproduce the slice-propagated pipeline of Cai et al. 2018b with no expert volumes. The strongest reported numbers are on DeepLesion: 76.4% mean 3D Dice from Cai et al. 2018b and 76% from Cai et al. 2018a. On the lymph node set, Cai et al. 2018a reports 70.2% Dice. Our reproduction must land near these numbers before any comparison is trusted.
Strongest competitor. This baseline gives Cai's pipeline the same expert volumes, in two ways. The first trains jointly on expert masks and unweighted propagated pseudo-labels. The second weights weak volumes per instance using the gradient-alignment indicator of Label-Efficient Hybrid-Supervised Learning, extended to 3D. This is the fairest test of our central claim, because it uses the same information with instance-level rather than voxel-level weighting. No reported 3D lesion result was found on Lacuna for this instance-weighting method; its 2D figure is 82.56% polyp Dice at 10% strong.
Standard self-training. Train on expert volumes, predict the weak volumes, and retrain on confident predictions with RECIST constraints. This is the generic recipe most practitioners would try first.
Ablations. Four ablations isolate the key ingredient:
- uniform $r=1$;
- a hand-tuned exponential decay in slice distance alone, replacing learned $r$;
- learned $r$ without the boundary term;
- the boundary term applied everywhere rather than only on reliable bands.
What counts as winning. Beating the strongest competitor means a statistically significant improvement on held-out expert volumes in both mean Dice and average symmetric surface distance (ASSD, mean distance between predicted and true surfaces). The improvement should concentrate in the far-from-anchor and low-contrast strata. Dice parity combined with lower ASSD in those strata still counts.
Why not just…?
Why not just train nnU-Net on the expert subset and self-train on the rest? This is the most serious threat. If confidence thresholds already pick out reliable voxels, then learned reliability adds little. The difference is that network confidence is known to be overconfident exactly where low-contrast boundaries mislead it; the dense-CRF recall collapse in Co-Segmentation is one symptom of trusting image-driven certainty at faint edges. Our reliability is checked against expert truth. The experiment settles this directly: we compare calibration curves of model confidence and of $r$ on held-out expert volumes, alongside stratified accuracy. If self-training matches us, we will report that.
Why not just weight whole weak cases, as the instance-indicator method does? One scalar cannot separate a correct RECIST-slice core from wrong distant slices, and Cai et al. document exactly this pattern of within-lesion decay. The 3D instance-weighted baseline is run head to head. If it matches voxel weighting, the per-voxel claim fails.
Why not just smooth 2D predictions in 3D, or prompt a 2D foundation model with RECIST boxes? Roth et al. found that 3D Gaussian smoothing lifted pancreas Dice only from 56% to 68%, and still failed when boundaries were faint. Work on 2D vision and promptable models, including ProMISe and Wolf et al., reports weak out-of-plane reasoning. We include smoothed 2D predictions as a cheap reference row. We do not expect it to compete on far-slice strata.
Why not just use a fixed distance-decay weight? This may well be enough, and if so the method simplifies. The hand-tuned decay ablation tests it directly. Learned $r$ is justified only if contrast and boundary-distance features improve far-slice ASSD beyond what slice distance alone delivers.
Proposed Experiments
Controlled experiments with simulated RECIST. The main experiments use datasets with full 3D masks, so RECIST marks can be simulated and the strong fraction controlled exactly:
- the lymph node CT set (984 nodes) from Cai et al.;
- the MSD pancreas task, where the Decathlon reports only 0.21 Dice on pancreas tumors;
- KiTS kidney tumors.
The lymph node set's 0.8-point gap between weak and full supervision leaves little Dice headroom, so its value is in surface distance and per-slice behaviour. Pancreas and kidney tumors carry the larger accuracy claim.
Real RECIST marks. DeepLesion then tests transfer to genuine clinical marks. The evaluation follows the 2D manual-mask test set and the 3D evaluation protocol of Cai et al., to the extent it can be reproduced.
Simulation realism. Simulated marks are cleaner than real ones. To probe the effect, we perturb endpoints and occasionally pick a non-maximal slice. We report results on both clean and perturbed simulated marks.
Decisive checks. Three checks decide whether the claim holds.
- Dice and ASSD against distance from the RECIST slice, in millimetres. The claim predicts a flatter curve than both purely weak propagation and instance weighting.
- Strong-fraction sweep. We train at 0, 2, 5, 10, 20, and 100% expert volumes. This shows how many expert volumes the reliability model needs, and whether learned reliability helps most at small fractions, as expected.
- Recall in low-contrast strata. Lesions are binned by measured boundary contrast, and recall is compared against the boundary-loss-everywhere ablation and dense-CRF post-processing. This tests whether gating boundary supervision by reliability avoids the recall collapse reported for CRFs.
If all three show no gain over the decay and self-training baselines, the conclusion is that weak-label errors in 3D are predictable from slice distance alone. That is a useful negative result for anyone planning annotation budgets.
References
- Accurate Weakly Supervised Deep Lesion Segmentation on CT Scans: Self-Paced 3D Mask Generation from RECIST
- Accurate Weakly-Supervised Deep Lesion Segmentation Using Large-Scale Clinical Annotations: Slice-Propagated 3D Mask Generation from 2D RECIST
- Label-Efficient Hybrid-Supervised Learning for Medical Image Segmentation
- Constrained-CNN losses for weakly supervised segmentation
- Every Annotation Counts: Multi-Label Deep Supervision for Medical Image Segmentation
- PnPNet: Pull-and-Push Networks for Volumetric Segmentation with Boundary Confusion
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- Weakly Supervised Lesion Co-Segmentation on Ct Scans
- Hybrid Transformer-Mamba Architecture for Weakly Supervised Volumetric Medical Segmentation
- Deep convolutional networks for pancreas segmentation in CT imaging
- ProMISe: Promptable Medical Image Segmentation using SAM
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
- The Medical Segmentation Decathlon