Telling Shadows from Disagreement in Weakly Labeled Ultrasound

When a model segments a lesion in an ultrasound image, it is uncertain at the boundary for two very different reasons. Sometimes the sound never reached the tissue. A calcification or dense mass reflects the beam and leaves a dark "acoustic shadow" behind it, so the boundary is literally not in the image. At other times the tissue is visible, but experts disagree about where the edge is. Current methods merge these into one uncertainty number. That includes the cheap, weakly supervised methods that learn from a few clicks instead of hand-traced masks. We propose a weakly supervised segmentation method for ultrasound that keeps the two apart. It reads an estimated map of where the beam actually reached from the image itself. It uses that map to stop click-to-mask propagation from latching onto shadow edges. And it learns the spread of expert opinion only where the tissue was actually imaged.

The practical payoff is that the two kinds of uncertainty call for different actions. Uncertainty from missing signal can be fixed by re-scanning from another angle. Uncertainty from expert disagreement can only be settled by adjudication. The enabling observation is economic. Multiple experts tracing full masks is unaffordable, which is the annotation bottleneck. Multiple experts each placing four extreme points (the topmost, bottommost, leftmost and rightmost points of the lesion) takes seconds per image. Weak supervision therefore makes multi-rater data cheap enough to model observer variability at all.

Core Claim

The claim has three parts. First, a per-pixel estimate of whether the region was insonified (whether sound reached it), computed along the probe's scanlines, should gate how sparse clicks are propagated into dense training masks. In imaged regions, image edges are trusted. In shadowed regions, edges are distrusted and the contour is completed from shape and from the extreme-point constraints. Second, training on several raters' extreme points lets the model learn a boundary distribution whose spread in imaged regions reflects genuine observer disagreement. Third, the resulting split into a "physics" uncertainty and an "observer" uncertainty is empirically valid. The physics part should shrink when a better-insonified view of the same anatomy is supplied; the observer part should not.

What Prior Work Does and Leaves Open

The method we build on is UA-EPT. It prompts SAM2, a general-purpose promptable segmentation model, with a box derived from four extreme points. It then traces boundaries between the points with Dijkstra shortest paths over Sobel gradients and Monte Carlo dropout uncertainty. Finally, it trains with a consistency loss that down-weights uncertain pixels. It reaches 67.70% IoU on BUSI, above its fully supervised baseline of 66.71%. It has two blind spots that matter here. It assumes one rater's points are correct. And because shadowed regions are high-uncertainty, its loss largely stops learning there, while Sobel paths can be pulled toward the vertical edges of shadows.

Learning to Segment Medical Images with Scribble-Supervision Alone supplies a similar expectation-maximization-style recipe on MRI, combining random walks, dense CRFs (conditional random fields, which smooth labels using pixel intensity similarity) and dropout gating. It loses only 2.9 to 4.5 Dice points against full supervision. It constrains our design, because its intensity-continuity assumptions are exactly what speckle and shadowing break.

The physical side comes from two papers. Weakly Supervised Estimation of Shadow Confidence Maps in Fetal Ultrasound Imaging shows that a dense shadow-confidence map can be learned from image-level labels plus a few coarse tracings. It runs at 0.0353 s per frame and agrees with consensus (Dice 0.7167) better than humans agree with each other (Dice 0.5443, ICC 0.51). We reuse it as the source of the insonification map. It never segments the anatomy behind the shadow, which is what we add. SSNet shows that a loss coupling a bone surface to its distal shadow is a useful physical constraint. It needs dense masks for both structures and a single ground truth. We borrow the idea that shadows are caused by something proximal along the scanline, without its supervision cost.

The observer side comes from Pionono, which gives each rater a Gaussian in a latent space whose mean captures bias and whose variance captures inconsistency, and from variational inference for inter-observer variability. Both need dense masks from every rater, and both were built on histopathology and MRI. They supply our rater model, applied to points instead of masks.

Label Uncertainty for Ultrasound Segmentation motivates the separation directly. Radiologists' per-pixel confidence, thresholded at 60%, gave the best downstream lung-ultrasound outcome prediction. But that confidence lumps artifact and subjectivity together, and thresholding discards exactly the shadowed margins. DiFF argues that ground truth should be whatever contour best serves diagnosis, with up to +2.51% AUC on thyroid ultrasound (TNMIX). It needs full multi-rater masks plus diagnoses. We treat its diagnosis-first view as a later evaluation lens, not as our training target.

Research Program

The pipeline has three stages.

The first stage estimates an insonification map. We train the shadow-confidence network of Weakly Supervised Estimation of Shadow Confidence Maps on the target modality from image-level shadow / no-shadow labels and a few dozen coarse tracings. We then re-express its output along scanline geometry: columns for linear probes, rays from the apex for curvilinear probes. Along each ray we enforce one physical constraint. Confidence may drop sharply only after a bright reflector and must stay low distally, which follows SSNet's proximal-cause logic. The output is $s(p) \in [0,1]$, the probability that pixel $p$ received signal.

The second stage turns clicks into masks in a physics-gated way. Each rater $r$ supplies four extreme points per image. For each rater we trace a boundary between that rater's points with a shortest-path cost

$$C(p) = s(p)\,\big(1 - g_{\perp}(p)\big) + \big(1 - s(p)\big)\,\lambda\,\kappa(p),$$

where $g_{\perp}(p)$ is the normalized image gradient after removing its component parallel to the local scanline direction, so that shadow side-walls stop looking like tissue edges. $\kappa(p)$ is a curvature penalty that favors smooth continuation of the contour. $\lambda$ trades the two off. In imaged regions the path follows real edges. In shadowed regions it completes the shape smoothly between the constraints. The SAM2 initialization from UA-EPT is kept, so the only change is the cost and its gating.

The third stage is a rater-aware model with two uncertainty outputs. A segmentation network with a Pionono-style rater latent is trained on the per-rater pseudo-masks. The ordinary loss applies only where $s(p)$ is high. In shadowed pixels the network is supervised by an extreme-point box-alignment loss and a shape-completion head, instead of being silenced. Observer uncertainty is the spread of predictions across rater latents. It is reported only where $s(p)$ is high, because that is the only place disagreement can be about what was seen. Physics uncertainty is $(1 - s(p))$ multiplied by the spread of the completion head under dropout. At deployment the model produces three things: a single contour from the mean rater latent, and two separate maps.

Proposed Experiments

Data. BUSI and UNS are the main weakly supervised benchmarks, because UA-EPT reports on both. On BUSI, malignant lesions frequently cast posterior shadows. For observer variability we use the TNMIX thyroid ultrasound benchmark used by DiFF, if its multi-rater masks can be obtained. Each rater's mask yields that rater's extreme points, which lets us train on simulated multi-rater clicks and evaluate against the real multi-rater masks. In parallel we collect real extreme points from three to five sonographers on a BUSI subset. This is cheap because four clicks per image take seconds, and it avoids the gap between simulated and real clicks. For the attribution test we use EchoNet-Dynamic and CAMUS, where apical dropout varies from frame to frame and from view to view.

Metrics. We report IoU and Dice against the official masks for comparability with UA-EPT. Against multi-rater references we report continuous Dice, the metric used on QUBIQ by variational inference for inter-observer variability. We also report boundary error restricted to contour segments inside estimated shadows, which isolates the claimed gain. Finally, we report the correlation between predicted observer spread and actual inter-rater spread, computed separately on imaged and shadowed boundary segments.

Baselines

Naive reference. The do-nothing reference is SAM2 prompted with the box implied by a single rater's extreme points, with no refinement. UA-EPT's ablation already places this about 7.4 IoU points below its full pipeline on BUSI. We report it to anchor how much any refinement buys.

Strongest weakly supervised competitor. This is UA-EPT itself, which reports 67.70% IoU on BUSI and 67.39% on UNS. Its fully supervised references are 66.71% and 67.54%. We rerun it from its published recipe on identical splits, with the same backbone, epochs and hyperparameter search budget as ours. To isolate the method from the extra data, we run it twice: once on one rater's points, and once with all raters' points averaged into a single set. Beating it means a statistically reliable gain in overall IoU on BUSI. We also require a larger reduction in shadow-segment boundary error, because that is where the mechanism should act. A gain overall with no gain in shadows would mean our story is wrong.

Semi-supervised reference. PH-Net reports 73.61% Dice on BUSI with 1/8 dense labels. It is run as a cost-matched comparison: dense masks on 1/8 of images versus extreme points from several raters on all images, at roughly matched annotation time.

Strongest observer-modeling competitor. This is a Pionono-style rater-latent network trained on dense multi-rater masks where they exist (TNMIX). It is an upper reference for observer calibration, since no reported result was found on Lacuna for Pionono on ultrasound.

Ablations of the key ingredient. Three ablations remove one piece at a time. The first replaces $s(p)$ with an all-ones map, which is plain gradient tracing. The second keeps $s(p)$ but simply drops shadowed pixels from the loss, which mimics confidence thresholding. The third uses one rater instead of several.

Why not just…?

Why not just generate one pseudo-mask per rater with UA-EPT and train Pionono on them? This is the real threat. It may match our overall IoU and observer calibration in imaged regions. The difference should show up in two places. One is shadow-segment boundary error, where UA-EPT's Sobel paths follow shadow side-walls for every rater alike, producing spurious "agreement" on a wrong boundary. The other is attribution, because that pipeline yields one undifferentiated spread. We run exactly this combination as a baseline. If it matches us on shadow boundaries too, the physics gating adds nothing, and the contribution reduces to the cheap multi-rater click protocol. We would report it that way.

Why not just detect shadows and exclude those pixels from the loss? Label Uncertainty for Ultrasound Segmentation shows that thresholding away low-confidence pixels shrinks and fragments the foreground and lowers IoU. In shadows, exclusion removes the very margins a clinician needs. The second ablation above is this alternative. Our prediction is that it matches us in imaged regions and loses on shadow segments and on contour completeness.

Why not just average the raters' extreme points? Averaging reduces rater noise in the pseudo-label but throws away the spread. A deterministic model trained on averaged points cannot report observer uncertainty, and the averaged-points run of UA-EPT tests whether it at least matches us on accuracy. On QUBIQ, averaging-style baselines trail latent observer models in continuous Dice (0.5042 for ESULR versus 0.5372), per variational inference for inter-observer variability. That suggests the spread carries signal, but it was measured on MRI and has to be rechecked here.

Decisive Checks

The attribution test decides whether the decomposition is real. On EchoNet-Dynamic and CAMUS we pair frames or views of the same heart in which apical dropout is present in one and absent in the other. We predict that physics uncertainty on the affected endocardial segment falls sharply in the well-imaged frame, while observer uncertainty there stays roughly constant. On the multi-rater BUSI subset, observer uncertainty should correlate with real inter-rater spread on imaged segments, and physics uncertainty should not. If both maps move together in these tests, the method has failed to separate the two sources, whatever its IoU. A secondary check uses TNMIX diagnoses to ask whether contours from the mean rater latent preserve DiFF-style diagnostic utility without DiFF's need for dense multi-rater masks. DiFF reports an AUC gain (+2.51%) rather than an absolute AUC, so the comparison must be rerun under matched conditions.