A 3D lesion segmenter labels each voxel of a CT or MRI volume as tumor or not. When it moves to a new hospital, it tends to miss tumor tissue, and nobody at that hospital can say by how much. Conformal risk control is a family of post-hoc methods that pick the model's decision threshold from a small set of labeled "calibration" cases so that a chosen error rate is guaranteed on future cases. Applied to 3D lesions, it can cap the fraction of tumor voxels the model misses (the false negative rate) at a target such as 10%. The guarantee holds only if the calibration cases come from the same distribution as the deployment cases, so each new site needs its own labeled calibration set. In current methods that means fully hand-contoured 3D tumor masks, which is the most expensive annotation in medical imaging.
This note proposes a way around that cost. To calibrate the missed-voxel rate, we do not need full masks. We need an unbiased estimate of that rate for each calibration volume. An annotator draws a loose 3D box around every lesion, which is detection-level work. The system then samples random voxels inside the boxes and asks the annotator a yes/no question for each: "is this voxel tumor?" The voxels answered "yes" are a uniform random sample of the true lesion. The fraction of them that the model scores below a candidate threshold is therefore an unbiased estimate of that volume's missed-voxel rate. We show that running conformal risk control on these estimates keeps the exact expected-risk guarantee on the true, unobserved miss rate. The same annotations also give a nearly exact count of false-positive volume. The result is that a new site can be recalibrated to a stated miss-rate bound for a small fraction of the annotator time that dense contouring would take.
Core Claim
The claim has two parts, one provable and one empirical.
The provable part concerns the loss. Let $s_i(v)$ be the model's lesion score for voxel $v$ in calibration volume $i$, and let the predicted mask at threshold $t$ be $\{v : s_i(v) \ge t\}$. The true loss $L_i(t)$ is the fraction of the volume's lesion voxels scored below $t$. This fraction grows as $t$ grows and lies between 0 and 1. From the $k_i$ "yes" voxels $p_{i1},\dots,p_{ik_i}$ we form the estimate
$$\hat L_i(t) = \frac{1}{k_i}\sum_{j=1}^{k_i} \mathbf{1}[s_i(p_{ij}) < t],$$
which is also nondecreasing in $t$, bounded by 1, and satisfies $\mathbb{E}[\hat L_i(t)\mid \text{volume}_i] = L_i(t)$ for every fixed $t$. We pick
$$\hat t = \sup\Big\{t : \tfrac{1}{n+1}\textstyle\sum_{i=1}^n \hat L_i(t) + \tfrac{1}{n+1} \le \alpha\Big\},$$
where $n$ is the number of calibration volumes and $\alpha$ is the target miss rate. The standard proof from Conformal Risk Control applies to the exchangeable pairs (volume, sampled clicks). It shows that the estimated loss on a new volume, $\hat L_{n+1}(\hat t)$, has expectation at most $\alpha$. The threshold $\hat t$ does not depend on the new volume's clicks, so that expectation equals $\mathbb{E}[L_{n+1}(\hat t)]$. The guarantee therefore transfers to the true miss rate. Noise from finite $k_i$ adds variance but no bias.
The empirical part is that, at equal annotator time, box-plus-click calibration beats dense-mask calibration. Two effects drive this. More calibration volumes shrink the $1/(n+1)$ slack in the rule above. More volumes also cover more of the site's case mix. Together these should produce a less conservative threshold, and so smaller predicted masks, at the same guaranteed miss rate.
Where This Sits
The mechanism comes from Conformal Risk Control, which bounds any bounded monotone expected loss (it reports 0.0987 empirical risk against a 0.10 target in polyp segmentation). It also comes from Conformal Lesion Segmentation for 3D Medical Images (CLS). CLS applies this idea to 3D CT and MRI across six lesion datasets and five backbones. On KiTS21 it cuts the missed-lesion fraction from about 0.53 at the default 0.5 threshold to about 0.10. CLS motivates this work and supplies its main baseline. It assumes dense calibration masks and explicitly lacks any handling of cross-site shift. We keep its guarantee and change what the calibration data must contain.
Two papers show why recalibration at each site is needed at all. Conformal Prediction Under Covariate Shift shows coverage falling from 90% to 82.2% under shift on tabular data. The FeTS challenge (Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge) shows segmenters with Dice "often above 0.9" that still fail at particular institutions, missing small enhancing foci. Its remedy for shift, reweighting calibration cases by a density ratio, cannot be computed over 3D image space. We avoid estimating that ratio by collecting cheap labels directly from the target site.
Two lines of work constrain the design by showing that label-free fixes are weak. Adapting Conformal Prediction to Distribution Shifts Without Labels adjusts conformal sets using prediction entropy, but recovers only 30% coverage (against a 90% target) on ImageNet-A. MAPSeg adapts 3D segmenters across sites without any target labels, but its pseudo-labels can reinforce misses on faint pathology. These results argue for using some target labels, as long as they are cheap.
Two papers use sparse human input under shift, but in different ways. Few Clicks Suffice (ATASeg) shows that 1–16 clicks per 2D image can adapt a street-scene model to near-supervised accuracy. CPATTA uses conformal scores to decide which 2D images to label for test-time adaptation. Both spend clicks on updating model weights and neither certifies a miss rate. We spend clicks on certifying the threshold, and we test whether doing both helps.
The closest non-conformal precedent is Semi-automated screening of biomedical citations for systematic reviews, which reaches 100% recall while halving review work in text screening. It shares our priority on recall but offers no spatial or statistical guarantee.
Research Program
We will build a query protocol and a calibration routine and prove the transfer result stated above. We then study four practical issues that the proof does not cover.
The first issue is how the rate is pooled. We define the per-volume miss rate over the union of all lesions in the volume. Points are sampled uniformly over the union of the boxes and non-lesion voxels are rejected, which keeps the "yes" points uniform over total lesion volume. We also define a per-lesion variant that weights each lesion equally, since clinicians fear missing a small lesion more than trimming a large one's margin.
The second issue is false positives. Because the boxes enclose every lesion, any predicted voxel outside the boxes is a false positive and can be counted exactly without further labels. Inside the boxes, the "no" answers estimate the false-positive fraction. This gives an almost label-exact measure of prediction bloat at any threshold, which Controlling False Positives in Image Segmentation via Conformal Prediction and CLS both identify as the price of guaranteeing recall. We report it next to every guarantee. As an extension, we test choosing between a few score functions, such as raw scores and connected-component-smoothed scores, on a held-out split of the calibration data.
The third issue is annotator reality. Boxes may cut off part of a lesion, and yes/no answers near fuzzy margins are noisy; FeTS reports 10.4% major errors even in dense reference labels. We measure how clipped boxes and flipped answers bias the estimate, and we test a simple guard of dilating each box by a fixed margin.
The fourth issue is the budget split. For a fixed annotator time, we choose how many volumes to calibrate on and how many clicks to request per volume. The estimate's variance is at most $1/(4k_i)$, from Bernoulli averaging. The slack term shrinks as $1/(n+1)$. The best split should therefore favor many volumes with modest $k$, and we measure where it actually falls.
Proposed Experiments
The primary testbed is the FeTS / BraTS 2021 multi-institution glioma data (The RSNA-ASNR-MICCAI BraTS 2021 Benchmark, FeTS). Institutions serve as sites. Each held-out institution is treated as a new deployment, and we evaluate the enhancing-tumor and tumor-core subregions, where FeTS documents site-specific misses. Segmenters are trained on the remaining institutions using nnU-Net, the self-configuring pipeline that won the Medical Segmentation Decathlon.
The secondary testbed is cross-dataset abdominal shift using the lesion datasets CLS already uses. Models trained on one tumor dataset are calibrated and tested on another of the same organ, such as MSD pancreas tumor to MDSC-Pancreas. We first reproduce CLS's in-distribution KiTS21 result to confirm our implementation. All boxes and clicks are simulated from the existing dense masks. The simulation adds realistic imperfections: boxes jittered and occasionally clipped, and yes/no flips concentrated near boundaries.
We also run a small reader study with radiologists. It measures time per box, time per yes/no answer, and time per dense 3D contour, so that simulated budgets can be expressed in real minutes.
The metrics are the realized true miss rate against the target $\alpha$ (mean and spread over many random calibration/test splits), the fraction of test volumes exceeding $\alpha$, false-positive volume per case, and prediction compactness. All are plotted against annotator minutes.
Baselines
The naive reference point is to deploy the source-trained segmenter at the new site with the fixed 0.5 threshold. A second reference point uses the same model trained with a false-negative-weighted Tversky loss and a lowered threshold chosen at the source site. These show how far off the miss rate is with no site-specific effort. The KiTS21 figure in CLS, about 0.53 at threshold 0.5, suggests the gap can be large even without shift.
The strongest competitor is CLS run with dense masks at the new site (Conformal Lesion Segmentation for 3D Medical Images). It delivers the same kind of guarantee using the most informative labels, and in-distribution it reaches about 0.10 miss rate on KiTS21. Our method must beat it or match it. We run it on the same trained model, the same site, and the same random test volumes. It receives the same annotator-minute budget, converted to a number of dense contours through the reader-study timings, and the same choice of $\alpha$. "Beating it" means one of two outcomes. Either we reach the target miss rate with a smaller per-case false-positive volume at equal minutes, or we reach it at comparable bloat with substantially fewer minutes. In both cases the realized miss rate must stay at or below $\alpha$.
A second guarantee-oriented baseline is source-calibrated CLS reused at the target site. A third is entropy-adjusted calibration in the style of Adapting Conformal Prediction to Distribution Shifts Without Labels, adapted to per-voxel scores. It uses no target labels and represents the "free" option.
Two ablations isolate the key ingredient. The first replaces uniform yes/no sampling with clicks placed where the model is most uncertain, an ATASeg-style active choice. This makes the clicks biased and should break the guarantee, showing that randomization, not click count, carries the result. The second uses the same click budget to adapt model weights in the manner of Few Clicks Suffice, or to split clicks between adaptation and calibration on disjoint volumes. This asks whether calibration or adaptation is the better use of scarce clicks.
Why not just…?
Why not just densely contour ten volumes at the new site and run CLS? This is the real threat. If a site sees few lesions per volume and a handful of contours already gives a tight threshold, box-plus-click calibration may buy little. The equal-minutes comparison against site-dense CLS settles this directly, and we will report where the two cost curves cross rather than assume our method wins. The argument for an advantage rests on the $1/(n+1)$ slack and on case-mix coverage. Both favor many cheaply labeled volumes over a few carefully labeled ones, particularly at small, heterogeneous sites like the failing FeTS institutions.
Why not just lower the threshold or train with a recall-weighted loss? A threshold tuned at one hospital has no guarantee at another, because score distributions drift across scanners. Segmentation Loss Odyssey and the Decathlon's 0.16 and 0.21 Dice on colon and pancreas tumors show that loss reweighting does not reliably fix hard lesions. The naive baselines measure exactly how far the realized miss rate drifts from the intended one at each held-out institution.
Why not use label-free shift correction, such as entropy scaling or unsupervised domain adaptation? Such methods cost no annotation, but they carry no guarantee. Their reported recovery under strong shift is partial: coverage on ImageNet-A goes only from 3% to 30% in Adapting Conformal Prediction to Distribution Shifts Without Labels. MAPSeg improves Dice but can lock in misses on faint lesions. We include both as baselines. If either keeps the realized miss rate under $\alpha$ across FeTS institutions, the case for spending annotation weakens, and we will say so.
Why not spend the clicks on adapting the model instead? Adaptation and certification are not substitutes. Clicks used to update weights make the model better but do not tell you its miss rate. The second ablation measures whether some clicks should go to adaptation on separate volumes before calibration.
Decisive Checks
The idea succeeds if three things hold. At held-out FeTS institutions, box-plus-click calibration keeps the realized true miss rate at or below $\alpha$ in expectation, while source-calibrated CLS and the naive threshold exceed it. At equal reader-measured minutes, it produces smaller false-positive volume than site-dense CLS. And the biased-click ablation visibly breaks the guarantee. The idea fails if simulated annotator noise at plausible levels pushes the realized miss rate above $\alpha$, or if site-dense CLS matches it at equal time across sites. Either outcome would show that the cheap estimate is not worth its fragility.
References
- Conformal Risk Control
- Conformal Lesion Segmentation for 3D Medical Images
- Conformal Prediction Under Covariate Shift.
- Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge
- Adapting Prediction Sets to Distribution Shifts Without Labels
- MAPSeg: Unified Unsupervised Domain Adaptation for Heterogeneous Medical Image Segmentation Based on 3D Masked Autoencoding and Pseudo-Labeling
- Few Clicks Suffice: Active Test-Time Adaptation for Semantic Segmentation
- Annotation-Efficient Active Test-Time Adaptation with Conformal Prediction
- Semi-automated screening of biomedical citations for systematic reviews
- Controlling False Positives in Image Segmentation via Conformal Prediction
- The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification
- The Medical Segmentation Decathlon
- Segmentation Loss Odyssey