In cancer imaging, radiologists already mark almost every tumor they report. Under the Response Evaluation Criteria in Solid Tumors (RECIST), they draw two crossing lines on a computed tomography (CT) slice: the lesion's longest diameter and the longest diameter perpendicular to it. These marks are free byproducts of routine care. Dense outlines of the tumor, which segmentation models need for training, are not free. Existing methods turn the crosshair into two shapes. The quadrilateral joining the four endpoints is almost surely inside the lesion. The smallest circle containing both lines almost surely encloses it. The hard question is where the true edge falls in the ring between the two shapes.
We propose to answer that question with a generative model rather than a discriminative one. We train an inpainting model, meaning a generator that fills a masked region so it agrees with its surroundings, using only tissue known to be lesion-free. We then ask it to "heal" the inside of the circle using the anatomy outside it. Pixels in the ring that the healed predictions explain well are background. Pixels that differ from every healed sample, measured against the spread across samples, are lesion. The output is a pseudo-mask that follows the lesion's actual shape rather than a convex template. A standard segmenter is then trained on these pseudo-masks. The approach is aimed at the irregular, lobulated and necrotic lesions where convex geometric bounds fail.
Core Claim
Our claim is that a lesion-free inpainting prior, restricted to the RECIST circle and read out as a per-pixel deviation normalized by sample spread, resolves the uncertain ring more accurately than the methods now used for it. Those methods are classical seed propagation, consistency between two discriminative networks, and intensity statistics. The gain should appear mainly on non-convex lesions and on boundary-distance metrics, not necessarily on average overlap.
Two lines of prior work motivate this claim. The first is RECIST weak supervision itself. Weakly Supervised Lesion Co-Segmentation on Ct Scans seeds GrabCut from the crosshair and trains a Siamese co-segmentation network across lesion pairs. RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training replaces GrabCut with the quadrilateral and circle shapes and co-trains two networks with a consistency loss in the ring. Neither paper puts any model of what healthy tissue looks like into the ring. The convexity assumption in the second paper is the known weak point.
The second line is counterfactual "healing" for weak segmentation. Pathology Segmentation using Distributional Differences to Images of Healthy Origin translates diseased MRI to healthy with a residual CycleGAN. COIN learns a perturbation that flips a classifier to "normal." What is Healthy? regenerates a healthy counterfactual of the whole image with diffusion, a generator that learns to turn noise into images by gradual denoising. All three must guess where the lesion is from image-level labels alone. The diffusion version regenerates the whole 64×64 frame, background included.
Our proposal differs from both lines at once. It uses the clinician's crosshair as a hard spatial bound, so the generator never has to find the lesion. It keeps every pixel outside the circle frozen, so background residuals are zero by construction. It replaces a single counterfactual with a calibrated test across several samples.
Research Program
The first component is the healthy inpainter. We train a mask-conditioned diffusion inpainter on CT slices at native in-plane resolution. Masks are disc- and ellipse-shaped regions, sized like RECIST circles, placed only where there is no annotated lesion. On DeepLesion and KiTS19 these are regions far from any bookmark, and organ regions from lesion-free scans. Known pixels are reinjected at every denoising step. This is the hard binary-mask constraint used by PHR-DIFF, and it follows the background freezing of InstantSwap. We deliberately avoid the bottleneck feature editing of Context-Preserving Dermoscopic Editing, whose own ablations report leakage outside the mask.
Because the inpainter never sees tumor during training, filling the circle should produce plausible organ tissue, vessels and organ edges rather than lesion. This plays the role that the healthy cohort plays in the counterfactual papers, without needing paired or matched cohorts.
The second component is the deviation score. For a slice $X$ with RECIST circle $C$ and quadrilateral $Q$, we draw $K$ healed samples $\hat X^{(1)},\dots,\hat X^{(K)}$, each inpainting all of $C$ from the context outside $C$. For each pixel $x$ in the ring $A = C \setminus Q$, we compute
$$ z(x) = \frac{\lvert X(x) - \mu(x) \rvert}{\sigma(x) + \epsilon}, $$
where $\mu(x)$ and $\sigma(x)$ are the mean and standard deviation of the $K$ samples at $x$ after light Gaussian smoothing, and $\epsilon$ is a small constant.
The pseudo-mask is $Q$ plus every pixel of $A$ with $z(x) > \tau$ that is connected to $Q$. Pixels with intermediate $z$ are marked "ignore" for the segmenter. Dividing by $\sigma$ is the key ingredient. Where healthy anatomy is itself uncertain, for example at a vessel the context only partly reveals, a large raw residual is discounted. Where healthy tissue is predictable, a small residual counts.
The third component is an optional shrinking loop. Once a first mask $M_1$ is estimated, we inpaint only $M_1$. The ring of $C$ outside $M_1$ then becomes extra context, which sharpens the healthy prediction near the edge. We recompute $z$ and iterate two or three times. This addresses the one obvious weakness of conditioning only on tissue outside $C$, which can be far from the true edge for elongated lesions.
The fourth component is downstream training. We train the same Swin-based segmenter used by Zhou et al. on the pseudo-masks. This keeps the backbone identical to the strongest baseline and isolates the source of supervision. The pseudo-masks can also be added as a third target alongside the label-space co-training loss. We report that combination separately, so we do not claim credit for co-training.
Baselines
The naive reference points are three fixed shapes used directly as the prediction: the quadrilateral $Q$, the circle $C$, and an ellipse fit to the two diameters. They need no learning and show how much any method gains over pure geometry.
The classical baseline is GrabCut seeded from the crosshair, followed by training the same segmenter on its output. Zhou et al. report a Dice score of 0.891 for this pipeline on KiTS19 in RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training.
The strongest existing method is Zhou et al.'s label-space co-training. It reports Dice 0.907 on KiTS19 with a Swin backbone, against a fully supervised upper bound of 0.912. On DeepLesion, the strongest is Agarwal et al.'s co-segmentation, at 90.3% mean Dice on 1,000 manually segmented lesions (Weakly Supervised Lesion Co-Segmentation on Ct Scans).
Fairness rules apply to these strong baselines. We re-run co-training ourselves with the same simulated RECIST protocol, splits, backbone, augmentation and tuning budget as our segmenter, and we also report the published numbers. The KiTS19 ceiling leaves only 0.005 Dice of headroom, so beating co-training on overall Dice is not the target. Success means matching it overall within seed variance while improving Dice on the bottom solidity tercile by at least three points. Solidity is lesion area divided by convex-hull area. Success also requires a lower 95th-percentile Hausdorff distance on that tercile.
A real competitor that is missing from the file is a promptable segmentation foundation model given the RECIST box. GazeRefine shows such models accept cheap spatial prompts, but no reported result was found on Lacuna for box-prompted foundation models on KiTS19 or DeepLesion. We run it zero-shot with the circle's bounding box, then again after fine-tuning on our pseudo-masks.
Ablations isolate each ingredient. In "ring statistics," a Gaussian model of intensity fit to the tissue just outside $C$ replaces the inpainter. In "raw residual," we drop the $\sigma$ normalization. In "single sample," $K=1$. In "generic inpainter," the inpainter is trained on all CT tissue, lesions included. "No shrinking loop" removes the iteration. Finally, a Sanchez-style whole-slice healthy counterfactual is restricted to $C$ only afterward, which tests whether hard context freezing matters.
Why not just…?
Why not just use the label-space co-training result, given it is already within 0.005 Dice of full supervision? On average it nearly closes the gap on KiTS19, and we do not expect to beat it there. Average Dice is dominated by roughly convex kidney tumors, where the quadrilateral and circle bracket the lesion tightly. The co-training paper's own construction assumes convexity. The open question is the irregular subset, where $Q$ cuts across necrotic margins. The solidity-stratified comparison settles this. If co-training already matches the fully supervised model on the low-solidity tercile, the proposal has no target and we would say so.
Why not just model the intensity of the surrounding tissue and classify ring pixels by likelihood? This is the most serious threat because it is cheap and uses the same idea of "healthy context explains background." Agarwal et al. report that GrabCut, an intensity-and-edge model, bleeds when lesion and parenchyma share intensity. A generative inpainter can predict structure that crosses the ring, such as a vessel or organ boundary, which a marginal intensity model cannot. Still, this is an empirical claim. The ring-statistics ablation tests it directly. If it comes within one Dice point of the full method on the irregular tercile, we drop the diffusion model and report the simpler method.
Why not just snap the circle to image edges with a dense conditional random field? Agarwal et al.'s ablation found that this refinement raised precision but sharply cut recall on low-contrast boundaries. The scribble work of ZScribbleSeg likewise found edge-based propagation gave unrealistic shapes without stronger priors. We include CRF refinement of $C$ as a baseline. Our expectation, which the experiment can refute, is that it trims the circle along the wrong edges for infiltrative lesions.
Why not just box-prompt a segmentation foundation model? This may be strong, and the file contains no number for it. We run it as described in the baselines. If zero-shot box prompting already wins on the irregular stratum, the proposal reduces to a pseudo-label source for fine-tuning such models. That is a smaller contribution, and we will measure it as such.
Proposed Experiments
We simulate RECIST crosshairs from ground-truth masks on KiTS19 following Zhou et al.'s protocol. On DeepLesion we use the real radiologist bookmarks, evaluated on the 1,000 masks used by Agarwal et al.
Metrics are Dice, the 95th-percentile Hausdorff distance, and ring-only Dice, which scores the pseudo-mask only inside $C \setminus Q$ and so measures exactly what the generator contributes. All three are stratified into terciles by ground-truth solidity. We also report how well $z$ is calibrated: the fraction of ring pixels with $z > \tau$ that are truly lesion, at several values of $\tau$.
The decisive check comes before any segmenter is trained. We compare ring-only Dice of our pseudo-masks with three alternatives on the low-solidity tercile: the co-training networks' predictions, the ring-statistics ablation, and GrabCut. If the healing pseudo-masks do not beat ring statistics there, the generative component is not earning its cost. A second check targets robustness to imprecise marks. We perturb the crosshair endpoints by a few pixels to mimic inter-reader variation and measure how fast each method degrades. Methods whose shape template comes directly from the endpoints should be more sensitive than one that reads the boundary from image content.
We also apply the same machinery to BraTS 2021 with simulated crosshairs. This shows how much a crosshair anchor adds over image-level healing. It is not a head-to-head comparison, since the supervision differs. What is Healthy? reports Dice 76.2 there with image-level labels only, at 64×64 resolution.
References
- Weakly Supervised Lesion Co-Segmentation on Ct Scans
- RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training
- Pathology Segmentation Using Distributional Differences to Images of Healthy Origin
- COIN: Counterfactual Inpainting for Weakly Supervised Semantic Segmentation for Medical Images
- What is Healthy? Generative Counterfactual Diffusion for Lesion Localization
- PHR-DIFF: Portrait Highlights Removal via Patch-aware Diffusion Model
- InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differences
- Context-Preserving Dermoscopic Editing: Mask-Guided Lesion-Aware Diffusion for Attribute Modification
- GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation
- ZScribbleSeg: Zen and the Art of Scribble Supervised Medical Image Segmentation