Certifying Organ Measurements from Rough Boundary Sketches

A segmentation model reports a pancreas volume or a tumor diameter, and a clinician needs to know how far off that number might be. The standard tool for this is conformal prediction, a distribution-free recipe that turns errors on a held-out calibration set into error bars with a coverage guarantee. It needs precise hand-drawn masks for every calibration case. Those masks are exactly what radiologists cannot afford to produce. They also have to be redrawn whenever the model moves to a new hospital, because a scanner or population shift breaks the guarantee.

This note proposes calibrating instead from "bracketing" annotations. For each calibration case the annotator draws a loose inner contour that is surely inside the organ and a loose outer contour that surely contains it, which takes seconds rather than minutes. The two contours bound the true measurement from both sides. A conformal score built on the worst case within that bracket keeps the coverage guarantee exactly. The price is wider intervals, and the width grows with how ambiguous the boundary really is. The practical payoff is that a hospital could re-certify a deployed segmentation model locally, cheaply, and in the very places where boundaries are faint and anatomy is irregular.

Core Claim

Low-contrast boundaries and anatomical variability are usually treated as obstacles to accurate masks. We treat them as the quantity to be measured. The claim has two parts. First, calibration with interval-censored labels (labels known only to lie between two bounds) gives valid error bars on clinical measurements without any dense calibration mask. Second, at a matched annotator-time budget, these intervals are competitive with, or narrower than, those from a smaller set of dense masks, because many cheap cases beat few expensive ones when the quantile must be estimated from small samples.

Formally, let $\hat m(x)$ be the model's measurement for scan $x$, such as volume, key-slice area, or long-axis diameter. Let $m^*$ be the true value, and let $[\ell, u]$ be the measurements computed from the inner and outer sketches, so that $\ell \le m^* \le u$. Define the bracket score

$$s(x) = \max\big(\hat m(x) - \ell,\; u - \hat m(x)\big),$$

which upper-bounds $|\hat m(x) - m^*|$ for every true value consistent with the sketch. Let $\hat q$ be the usual finite-sample conformal quantile of $s$ over $n$ calibration cases. Then the interval $\hat m(x) \pm \hat q$ contains the true measurement with probability at least $1-\alpha$ under exchangeability, since $\{s \le \hat q\}$ implies $\{|\hat m - m^*| \le \hat q\}$. The guarantee also covers any expert whose boundary falls inside the bracket. This matters because, as Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle shows, experts disagree widely on low-contrast edges, and a single rater's mask is not privileged.

Positioning

The closest methods certify segmentation-derived measurements but assume dense masks. COMPASS calibrates in the network's internal feature space and gets conformal bounds on area and volume errors roughly 50% narrower than split conformal on Kvasir at 90% coverage. CARE builds confidence intervals for ratio biomarkers and shows that network miscalibration dominates their error. Both need pixel-accurate ground truth on the calibration set, and both lose their guarantees under the scanner and site shifts that the file identifies as the heart of the clinical validation gap. We keep their goal, certified measurement error, and remove the dense-label requirement. COMPASS's feature-space score becomes a drop-in point predictor inside our bracket score rather than a competitor.

The bracketing annotation comes from the weak-supervision line of work, where it was used only for training. EAUWSeg introduced inner/outer bounded polygons with an explicit "gray zone." It trained a skin-lesion segmenter at 86.60% Dice on ISIC2017, above full supervision, using 10 seconds of annotation per image instead of 55. RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training derives an inscribed quadrilateral and a circumscribed circle from the two diameters of RECIST, the routine tumor measurement protocol, and reaches 0.907 Dice on KiTS19. Its inner bound is valid only for convex lesions, which is a constraint we take seriously. We therefore use RECIST-derived brackets only as a cheap, possibly invalid tier and measure how often convexity fails. Human-drawn brackets make no shape assumption and so handle concave or infiltrative anatomy.

Neither paper uses the gray zone for calibration. The negative results on seed-growing refinement also constrain us. GrabCut and dense CRFs (conditional random fields, a pixel-smoothing post-process) leak or lose recall at low-contrast borders, as Agarwal et al. and the co-training paper report. This is why we refuse to treat any refined pseudo-mask as calibration truth.

The shift motivation comes from Label-free estimation of clinically relevant performance metrics under distribution shifts. There, confidence-based monitors reach about 0.007 accuracy-estimation error in-distribution, degrade to about 0.05 out-of-distribution, and break down under prevalence shift. If label-free monitoring cannot be trusted at a new site, the remaining option is cheap local labels, and this proposal makes those labels cheap.

Research Program

We fix segmentation backbones and study calibration only. The backbones are a 3D nnU-Net and PanSAM, where the latter is PanSAM, a SAM adaptation that generates its own prompts and reports 87.01% Dice on NIH Pancreas and 83.20% zero-shot on AbdomenCT-1K. The measurements are those clinicians actually use. They are organ and tumor volume, key-slice area, and long-axis diameter on pancreas and kidney tumor. They also include a ratio biomarker (tumor-to-kidney volume on KiTS23, and necrotic fraction on BraTS21) to connect with CARE.

Key-slice area and diameter get exact guarantees, because the brackets are drawn on that slice. Volume is harder, because sketches on every $k$-th slice do not strictly bound the volume between them. We make the needed assumption explicit: we bound between-slice cross-sectional area using the brackets on neighboring slices plus a slack term. We then measure empirically, on densely labeled volumes, how often the slack is violated as $k$ grows. This is the main place where anatomical variability enters. Rectum-like or pancreatic shapes that change quickly between slices should need smaller $k$, and we report the annotation cost this implies.

The guaranteed intervals are conservative by construction. We therefore add a second, explicitly non-guaranteed tightening. A small dense subset is used to learn where inside $[\ell, u]$ the truth typically falls, as a function of gray-zone width. The bracket score is then scaled accordingly, and a held-out dense set checks its coverage. We report the guaranteed and tightened versions separately and never conflate them.

To ground annotation time in CT, we run a timing study with radiologists who draw brackets versus full contours on pancreas and kidney slices. EAUWSeg's 10-versus-55-second figure is for 2D dermoscopy. No reported result was found on Lacuna for CT bracket-annotation time, and every matched-budget comparison below depends on this measurement.

Baselines

The naive reference point is the model's own uncertainty. The interval is the spread of measured volume across Monte Carlo dropout samples, or across thresholds of the softmax map, and no calibration data is used. We expect it to under-cover, and CARE's finding that miscalibration dominates biomarker error predicts that. Any method that fails to beat its coverage at the nominal 90% is not worth deploying.

The strongest competitor is dense-label conformal calibration at equal annotator time. We run split conformal and COMPASS-J (COMPASS), and CARE for the ratio task, using as many densely masked calibration cases as the measured time budget allows. That is roughly one dense case per several bracketed ones, depending on the timing study. Both arms use the same backbone, the same held-out test scans, and the same target coverage, and each gets identical hyperparameter search over score type. On 2D Kvasir the file reports COMPASS-J at about half the interval width of split conformal at 90% coverage. No reported result was found on Lacuna for COMPASS or CARE interval widths on 3D pancreas or KiTS volumes, so we establish those numbers ourselves. Beating this baseline means one of two things at equal time: equal coverage with a narrower median interval, or, at small budgets, reaching nominal coverage where the dense arm's quantile is too noisy to do so reliably, as measured over repeated calibration draws.

A third baseline uses pseudo-labels as truth. We compute conformal intervals with masks from label-space co-training or GrabCut refinement of the same sketches as calibration ground truth. It uses the same information as our method but a biased label, so it isolates whether respecting the bracket is what buys validity.

The key ablation replaces the worst-case bracket score with the midpoint label $(\ell+u)/2$. That is the obvious shortcut, and we expect it to under-cover precisely on low-contrast cases where the truth sits near one bound.

Why not just…?

Why not just densely label a small calibration set? This is the real threat. Split conformal needs only a few hundred cases for 90% coverage, and dense masks give tight scores. The answer is empirical, and the matched-budget experiment settles it. If dense calibration wins at every budget, the proposal's contribution shrinks to the multi-rater and shift results, and we will say so. We expect brackets to win at small budgets, because the variance of a conformal quantile shrinks with sample size. We also expect them to win on the most ambiguous organs, where a single dense mask is itself one rater's guess, which the file notes is an acknowledged limitation of The Medical Segmentation Decathlon.

Why not just use the model's own uncertainty, or train it to predict error bars? Weakly or even fully supervised networks are miscalibrated. CARE shows miscalibration is the dominant error source in biomarkers, and COMPASS reports post-hoc feature calibration beating end-to-end conformalized quantile regression. The naive baseline above measures how badly self-reported uncertainty covers under site shift.

Why not just refine the sketches into masks with GrabCut or co-training and calibrate on those? Refinement leaks at low-contrast borders. Agarwal et al. saw dense CRFs sharply cut recall, and GrabCut trailed co-training on KiTS19 (0.891 vs 0.907 Dice). Any systematic pseudo-label bias passes directly into coverage error. The pseudo-label baseline tests this directly. If its coverage holds, the simpler route is fine for that organ.

Why not just monitor performance without labels at the new site? The label-free estimators in Label-free estimation... degrade out-of-distribution and fail under prevalence shift, which is routine across hospitals. We include confidence-thresholding volume-error estimation (average-thresholded-confidence style) as a comparator in the shift experiment.

Proposed Experiments

The first experiment is validity with simulated brackets. On NIH Pancreas, MSD Pancreas, and KiTS19/23, we generate inner and outer contours from dense masks by sampling realistic margins, which are fitted to the brackets our radiologists draw in the timing study. We then check that bracket-score intervals achieve 90% coverage on measurements from the true masks across 200 random calibration splits. We report coverage and median width stratified by gray-zone width and by shape irregularity (convexity deficit). The decisive check is that coverage holds on the most concave decile. On that decile the RECIST quadrilateral tier should violate its inner bound, and we report the violation rate directly.

The second experiment is matched-budget efficiency. It traces interval width against annotator minutes for the bracket, dense split-conformal, and dense COMPASS-J arms. The proposal is supported if the bracket curve lies below the dense curves at the budgets a hospital would realistically spend, tens of minutes to a few hours.

The third experiment is multi-rater validity. On REFUGE-2, which has several experts per image, we calibrate once from brackets. We then check coverage of the cup-area measurement against each rater separately. Dense single-rater calibration is expected to under-cover the other raters, and bracket calibration should not.

The fourth and decisive clinical experiment is site shift. We calibrate the pancreas model on NIH Pancreas and deploy it on AbdomenCT-1K's twelve centers. We first measure how far source-calibrated intervals fall below nominal coverage per center. We then recalibrate each center with $n$ bracketed cases from that center and report the $n$ and annotator minutes needed to restore 90% coverage, compared against dense recalibration and against label-free monitoring. If a few hours of bracketing per site restores certified error bars that dense relabeling would take days to provide, the proposal has closed a concrete piece of the clinical validation gap.