Hospitals already store millions of tumor measurements made with RECIST (Response Evaluation Criteria in Solid Tumors). Radiologists draw the longest diameter of a lesion on one CT slice and a perpendicular short diameter. Weakly supervised segmentation methods turn these two lines into rough regions, such as an inscribed quadrilateral and an enclosing circle, and train a network to fill in the outline. They largely ignore a simpler fact: the four endpoints of those lines are points the radiologist placed on the lesion boundary. Every measured lesion, including rare lesion types and faint, low-contrast ones, therefore comes with a few exact boundary labels at no cost. Current methods report one aggregate Dice score over all lesions, which hides where they fail.
This note proposes to use those endpoints as boundary labels in a separate second training stage. The idea comes from long-tailed classification, where bias toward common classes sits mostly in the final decision layer rather than in the features. Our hypothesis is that weakly supervised lesion segmenters show the same pattern for boundaries. They place the boundary where it would be for common, high-contrast lesions, so rare or faint lesions are systematically over- or under-segmented. If so, a small correction trained on endpoints with equal weight per lesion group should fix much of the tail error without retraining the network.
Core Claim
The claim has two parts, and both can be tested. The first is a diagnosis. In a strong RECIST-supervised segmenter, the signed distance from the predicted contour to the radiologist's endpoints has a systematic, group-dependent mean. A group here is a lesion appearance cluster or a contrast level. A bias of this kind would show up as a consistent direction of error, as opposed to scatter around zero.
The second part is a remedy. A recalibration with few parameters can remove that bias. It is fit after the network is trained, uses only endpoint and quadrilateral/circle constraints, and samples lesion groups evenly. It should improve Dice on rare and low-contrast strata while leaving aggregate Dice unchanged. Few parameters are needed because each group requires only a shift in where the boundary sits, and a group with a few dozen lesions supplies a hundred or more exact boundary points.
The closest prior work sets up this distinction. RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training uses the RECIST endpoints only as vertices of an under-segmentation region. It enforces consistency inside the uncertain ring between that region and an over-segmentation circle. It never asks the contour to pass through the endpoints, and it was tested on a single lesion type. Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss confines an active-contour loss to the RECIST region but treats all lesions as one class. PAT: Pixel-wise Adaptive Training for long-tailed segmentation reweights uncertain boundary pixels of rare classes, but it needs dense masks and was evaluated on street and indoor scenes. Our proposal differs from all three in two ways: it supervises boundary location directly from sparse clinical points, and it moves balancing into a separate stage after training.
Research Program
Stage 1 (base model). We train a 2D RECIST-supervised segmenter on DeepLesion. The base model is AHRNet with the Regional Level Set loss, reimplemented following Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss. The training set follows the natural lesion distribution, so the features learn from all data.
The encoder starts from one of two initializations:
- Natural-image weights. ImageNet initialization.
- In-domain weights. Self-supervised pre-training on unlabeled DeepLesion slices with the superpixel-guided contrastive objective of SuperCL: Superpixel Guided Contrastive Learning for Medical Image Segmentation Pre-Training. SuperCL contrasts pixels inside a superpixel against pixels outside it.
Domain pre-training is a controlled factor here, not the main contribution. It lets us ask whether better in-domain features already remove the boundary bias or leave it in the decision stage.
Diagnosis. Before building anything new, we measure the bias on the trained base model. For each training lesion we record the predicted signed distance at its four endpoints. We group lesions in two ways:
- by the appearance pseudo-categories obtained by clustering, as in DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning and Weakly Supervised Lesion Co-Segmentation on Ct Scans;
- by a local contrast score, defined as the intensity difference across each endpoint along the diameter direction. This score is a measurable stand-in for boundary ambiguity.
The project continues only if per-group mean offsets differ reliably from zero and from each other across random seeds. If the errors turn out to be unbiased scatter, the second stage has nothing to correct, and we report that as a negative result.
Stage 2 (decoupled recalibration). We freeze the stage-1 network and fit a small head on top of the output logits $s(p)$ at pixel $p$. The head has a per-group scale and offset plus a shared 1×1 convolutional residual on the last feature map. The group comes from the cluster assignment, which can be predicted from the features at test time. The objective is
$$ \mathcal{L}_2=\frac{1}{G}\sum_{g=1}^{G}\frac{1}{|E_g|}\sum_{p\in E_g}\max\big(0,\,|s(p)|-\tau\big)\;+\;\lambda\,\mathcal{L}_{QC}. $$
The terms are defined as follows:
- $G$ is the number of lesion groups.
- $E_g$ is the set of RECIST endpoints for lesions in group $g$.
- $\tau$ is a tolerance that absorbs one or two pixels of annotator imprecision.
- $\mathcal{L}_{QC}$ is a binary cross-entropy that marks pixels inside the inscribed quadrilateral as foreground and pixels outside the circumscribed circle as background.
- $\lambda$ balances the two terms.
Averaging over groups first, rather than over all endpoints, gives a rare cluster the same weight as liver or lung lesions.
Baselines
Naive reference. The do-nothing point is the stage-1 model used as is, with a 0.5 threshold. Every gain is measured against it with the same encoder initialization.
Strongest existing method. The main competitor is AHRNet with the Regional Level Set loss, which is the best weakly supervised result on DeepLesion in the file. The authors report "Dice score of 92.6%" on DeepLesion and "88.2% Dice score" on an external set of 470 lesions, surpassing nnU-Net by 2.6% and 2.7% respectively. Weakly Supervised Lesion Co-Segmentation on Ct Scans reports a lower "mean Dice score of 90.3%".
We will reproduce AHRNet + RLS with the same preprocessing, the same split, and the same three seeds. Its quoted numbers serve only as a sanity anchor, because exact test-set comparability is not guaranteed. Since aggregate Dice is already near its ceiling, "beating it" means two things together:
- aggregate Dice stays within seed-to-seed spread of our reproduction;
- Dice on the rarest third of clusters and on the lowest-contrast quintile improves by more than that spread.
Label-space co-training. RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training is the strongest method that uses the endpoints implicitly. It reports a Dice score of 0.907 on KiTS19 kidney tumors, against 0.891 for GrabCut pseudo-masks and a fully supervised bound of 0.912. We run it on DeepLesion and on KiTS19 under the same tuning budget, measured as the number of hyperparameter trials.
Class-balanced training. This baseline retrains stage 1 end to end with group-balanced sampling. It is the standard non-decoupled answer to imbalance.
Ablations that isolate the key ingredients. The first adds the endpoint term to stage-1 training without freezing and without group balancing. This separates the value of endpoint supervision from the value of decoupling. The second runs stage 2 with endpoints averaged uniformly over lesions instead of over groups. This isolates the balancing.
Why not just…?
Why not just tune a per-group threshold on a validation set? This is a real threat. It is the zero-capacity version of our stage 2. We include it, fitting the thresholds on the same endpoints. If it matches the full head, the honest finding is that the boundary bias amounts to one scalar offset per group. That would still be new and useful: a cheap fix with no new network, plus a diagnosis that no prior DeepLesion paper reports.
Why not just reweight rare lesions during end-to-end training? The long-tailed chest X-ray study, Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study, found that rebalancing the final layer after normal training works better than rebalancing throughout. The reason given is that resampling during representation learning distorts the features. Whether this carries over to segmentation is unknown, so the class-balanced stage-1 baseline tests it directly. We also report its aggregate Dice, to show whether common lesions pay for the gains on rare ones.
Why not just prompt the Segment Anything Model with the RECIST endpoints? The Segment Anything Model (SAM) is a general-purpose segmentation model that accepts point or box prompts. Point-prompted SAM fits naturally here. However, CaPro: Curvilinear-aware Prompt Learning with Single Unlabeled Image for Cost-effective Curvilinear Structure Segmentation reports that SAM needs specialized adaptation for low-contrast medical structures. We found no reported result on Lacuna for SAM on DeepLesion RECIST segmentation, so we run SAM with endpoint and box prompts as a baseline and report it per contrast quintile.
SAM also needs the endpoints at inference. That makes it comparable to a second reference we include: snapping each test lesion's threshold to its own endpoints. Both apply only to lesions a radiologist has already measured. Our method has to work without test-time measurements, so these two serve as an upper reference, not as direct competitors.
Why not just generate GrabCut pseudo-masks and train nnU-Net? The file already shows this falls short. GrabCut masks reach 0.891 Dice against 0.907 for co-training on KiTS19 (RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training), and nnU-Net trails AHRNet + RLS by 2.6% on DeepLesion. We keep the GrabCut + nnU-Net pipeline as a baseline because its errors should be worst on exactly the low-contrast stratum we target.
Proposed Experiments
Data. The main benchmark is DeepLesion. We train on RECIST marks only and evaluate on its manually masked test set of 1,000 lesions from 500 patients. Where we can obtain it, we also evaluate on the external 470-lesion set used in Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss. KiTS19 tumors, with RECIST-style marks simulated from the masks, serve as a second domain on which RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training can be reproduced.
Metrics. We report:
- aggregate Dice;
- Dice per appearance cluster, summarized as the mean over the rarest third and the worst-group Dice;
- Dice per contrast quintile and per size tercile;
- the 95th-percentile boundary distance, which measures how far the worst parts of the contour stray.
Decisive checks. There are three.
- Bias existence. The diagnosis must show a group-dependent signed bias at the endpoints. Without it, the project stops.
- Endpoint agreement predicts boundary quality. On the 1,000 manual masks, a lesion's endpoint error after recalibration must track its true boundary error. Otherwise, matching four points may just bend the contour at those points while it still misses elsewhere along the boundary.
- Factorial test with pre-training. We cross the two encoder initializations with recalibration on and off. If in-domain pre-training removes most of the stage-2 gain, the bias lives in the features. If the gains add up, it lives in the decision stage. Either result tells practitioners whether to invest in domain pre-training or in cheap recalibration when lesions are rare and faint.
References
- RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training
- Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss
- PAT: Pixel-wise Adaptive Training for long-tailed segmentation
- SuperCL: Superpixel Guided Contrastive Learning for Medical Image Segmentation Pre-training
- DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning
- Weakly Supervised Lesion Co-Segmentation on Ct Scans
- Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study
- CaPro: Curvilinear-aware Prompt Learning with Single Unlabeled Image for Cost-effective Curvilinear Structure Segmentation