Clicks on the Contour: Training Boundary Heads from the Points Clinicians Already Mark

Medical segmentation models need pixel-perfect masks, and only specialists can draw them. The cheaper labels people use instead are bounding boxes, four "extreme point" clicks at the top, bottom, left and right of a lesion, or the RECIST lines radiologists already draw to measure tumors (a long and a short diameter, by clinical convention). The field treats these as rough hints about where the region is. Most of them are something more specific. Every extreme-point click and every RECIST endpoint lies exactly on the object's outline. A box touches the outline at least once on each of its four sides. Weak labels are therefore sparse samples of the boundary itself. In ultrasound and soft-tissue CT, the boundary is precisely where models fail: the lesion and the surrounding tissue have nearly the same intensity, so the mask leaks outward.

We propose to train a dedicated boundary-prediction branch directly on these few on-contour points. The branch is tied to the region mask by a consistency term that forces its outline to match the edge of the predicted mask, and it may report where along the contour it is unsure. The resulting model needs no offline pseudo-mask loop, no hand-crafted edge filter and no natural-image edge detector. Our claim is that this recovers margins in low-contrast images better than current pipelines that first build pseudo-masks and then train on them. We test this on breast and nerve ultrasound and on RECIST-annotated CT, at matched annotation cost.

Core Claim

Boundary-aware multi-task networks, meaning a shared encoder with one output branch for the region mask and another for its contour, reliably sharpen margins and split touching objects. The DCAN gland network (DCAN) and the liver-tumor network E2Net (E2Net) are examples. Every one of them derives its contour target from a full expert mask. Weakly supervised methods drop the boundary branch, as in the scribble-only work of Can et al. and the box-constraint work of Kervadec et al.. Others supervise it with signals that break in low contrast: an off-the-shelf natural-image edge detector in Boundary Perception Guidance, or a fixed Sobel gradient filter in CTO and UA-EPT.

We claim the boundary branch can be supervised from three sources that weak labels and the model already supply. The first is positive boundary targets at the on-contour points the annotator clicked. The second is the box geometry, enforced as inequality constraints on the mask. The third is a duality term that makes the boundary map agree with the spatial gradient of the mask, so each head teaches the other between the sparse points. The point targets also prevent the trivial solution in which both heads predict nothing, which RPCNet never faced because it was fully supervised.

Research Program

The model is a standard encoder–decoder with two heads. One outputs a foreground probability map $m$. The other outputs a boundary probability map $b$ and a per-pixel positive scale $\sigma$ that expresses how uncertain the boundary location is. Training minimizes

$$\mathcal{L} = \mathcal{L}_{\text{pts}}(b) + \lambda \sum_{x} \left[ \frac{\big(b(x) - g(m)(x)\big)^2}{\sigma(x)^2} + \log \sigma(x)^2 \right] + \mathcal{L}_{\text{box}}(m).$$

The terms are as follows. $\mathcal{L}_{\text{pts}}$ is a focal loss that pushes $b$ toward 1 in a small disk around each annotated on-contour point, and toward 0 outside the box. $g(m)$ is the normalized gradient magnitude of the mask, so the middle term is the duality term of RPCNet. Here it is weighted heteroscedastically, meaning by a learned per-pixel noise scale: along stretches with no image evidence, the model can inflate $\sigma$ and pay a $\log\sigma^2$ cost rather than snapping to speckle. $\mathcal{L}_{\text{box}}$ holds the emptiness, tightness and size inequalities of Kervadec et al., solved with their log-barrier penalty. $\lambda$ weights the duality term.

Each prior paper plays a different role here. Kervadec et al. and RPCNet supply method pieces we reuse. RecycleNet, which refines ambiguous decisions by feeding features back through the network, is a possible later extension; it is not part of the core claim.

The learned $\sigma$ replaces two existing mechanisms. One is the fixed-width morphological uncertainty band of Yeung et al., which needs full masks and uses the same width everywhere. The other is the 20-pass Monte Carlo Dropout uncertainty in UA-EPT and Can et al., which estimates uncertainty by running the network many times with random units switched off. Our estimate comes from a single forward pass.

Work proceeds in three stages. The first builds the label converters. Extreme points on BUSI and UNS map to four boundary points plus a box. RECIST lines on DeepLesion map to four endpoints plus the enclosing box. PROMISE12 boxes give tightness constraints only, which makes it a stress test with no explicit boundary points. The second stage trains and ablates the objective. The third analyzes where along the contour the gains come from. To do this, we bin ground-truth contour pixels by local intensity contrast and report boundary error per bin. This is the analysis that tests the "low-contrast" part of the claim.

Baselines

The naive reference trains the same network with the filled box or the extreme-point polygon treated as a full mask. It shows how much any method gains over taking the weak label literally.

The first constraint-only baseline is Kervadec et al.'s log-barrier box method (Bounding boxes for weakly supervised segmentation). It is the cleanest proposal-free competitor, and our method equals it with the boundary head and point terms removed, so the comparison isolates exactly what the boundary branch adds. The file reports no number for it on PROMISE12 or ATLAS ("no reported result found on Lacuna"), so we rerun it.

The strongest existing method on our main benchmarks is UA-EPT (UA-EPT). It turns extreme points into SAM2 box prompts, then refines with uncertainty-weighted Dijkstra path tracing (a shortest-path search) and a dual-branch network. It reports 67.70% IoU on BUSI and 67.39% IoU on UNS, against fully supervised references of 66.71% and 67.54%. On DeepLesion, the strongest reference is the RECIST co-segmentation of Agarwal et al., at 90.3% mean Dice.

We will report the published numbers and also rerun UA-EPT with our backbone, the same splits, the same epoch budget, and an equal number of hyperparameter trials for every method. This separates method gains from backbone gains. Because UA-EPT draws on SAM2, a large pretrained model, we also report a variant of ours that initializes from SAM2 pseudo-masks. That way the comparison is not confounded by access to the foundation model.

Beating UA-EPT means matching its IoU within noise while clearly lowering Hausdorff distance and boundary error in the low-contrast bins. It also means doing so in one training run rather than an offline refinement loop, measured in GPU-hours.

The ablations that isolate the key ingredient are as follows. We remove the separate boundary head and impose the point targets directly on $g(m)$. We remove the duality term. We fix $\sigma$ to a constant. Finally, we replace point supervision with Sobel-derived edges inside the box.

Why not just…?

Why not just prompt SAM2 with the box and train a boundary-aware network on its masks? This is the most serious threat. A DCAN-style network with Kervadec's boundary loss (Boundary loss), computed from SAM2 pseudo-mask distance maps, is easy to build. UA-EPT's ablation shows SAM2 adds about 7.4 points over a plain box baseline but still needs refinement to reach its final numbers. That suggests SAM2 masks are coarse at ultrasound margins, but it does not show that a boundary-aware student cannot fix them. We run this pipeline as a full baseline. If it matches our method in the low-contrast bins, the claim that on-contour points add value is false, and we will report that.

Why not just put the point targets on the mask's own gradient and skip the second head? This is plausible. Duality says the two should coincide, so the separate head might be redundant. The argument for keeping it is that a dedicated decoder shapes the shared encoder, pushing it to keep fine spatial features, which is the mechanism behind DCAN and E2Net. This is an empirical question, and the first ablation answers it directly. If the single-head variant matches, we will adopt it, since it is simpler.

Why not just snap predictions to edges with a DenseCRF or a Sobel filter? The file's evidence argues against both. In Agarwal et al., adding a DenseCRF on DeepLesion cut recall sharply. Can et al. report that DenseCRF is fragile to its bandwidth settings. CTO notes that a fixed Sobel filter confuses speckle and texture with real margins. Even so, we include CRF post-processing on the box baseline and the Sobel-edge ablation, so the contrast-binned analysis shows whether these failures actually appear on BUSI and UNS.

Proposed Experiments

The main experiments use BUSI and UNS with extreme-point labels, reporting IoU, Dice, 95th-percentile Hausdorff distance and boundary F-score at 2 and 5 pixel tolerance. These are stratified by contour contrast bin and, for BUSI, by benign versus malignant cases, since GlaS documents steep degradation on irregular malignant shapes. DeepLesion with RECIST labels is evaluated on the 1,000 expert masks. PROMISE12 with boxes alone checks whether the method degrades gracefully when no boundary points are given.

Three results would decide the question. The first is whether the gap between our method and the SAM2-pseudo-mask boundary-aware baseline widens in the lowest-contrast bins; if it does not, the core claim fails. The second is whether the learned $\sigma$ correlates with where annotators disagree. Expert disagreement is unavailable on these sets, so we use agreement between our prediction and SAM2 as a proxy, and the high-$\sigma$ regions should concentrate where boundary errors occur. The third is an equal-cost curve: Dice against annotation seconds, comparing extreme points, RECIST and a small fraction of full masks. This shows whether boundary-point supervision moves the cost–accuracy frontier or merely matches it.