Medical segmentation models increasingly try to learn how each expert draws boundaries rather than averaging experts into one "true" mask. Doing this requires many experts to contour the same scans, and that is expensive. Wireless engineers face a similar problem. Coordinated transmission promises large gains only if every node knows the channel perfectly, but learning the channel costs airtime, and the cost can consume the whole budget. Their standard remedy is the pilot: a short, known training signal that everyone measures. It is spent once per window over which the channel stays roughly stable, and all remaining resources carry data. We propose the same accounting for annotation. A small shared set of "pilot images" is contoured by every rater and used only to estimate each rater's drawing style. Every other image receives exactly one label from one rater. The pilots are refreshed when a rater's style drifts. The research question is how large the pilot set should be, and how often it must be renewed, so that a model trained under a fixed total label budget best reproduces the full expert panel.
In two sentences: rater style behaves like a channel that is stable over many images, so paying a small shared calibration cost and then collecting single labels may beat both full panels on few images and uncoordinated single labels on many. We want to test this and characterize the best split between calibration labels and coverage labels.
Core Claim
The claim is that at equal annotation cost, a pilot-calibrated protocol reproduces the multi-rater distribution better than either extreme, measured by Dice to each rater's held-out masks and by distribution agreement with the panel. One extreme is full-panel labeling of fewer images, the analogue of assuming perfect channel knowledge. The other is one label per image with no shared images, the analogue of blind transmission. We also claim that the best pilot fraction follows a predictable trade-off rather than being arbitrary.
To state the trade-off, let $B$ be the total number of contours affordable, $R$ the number of raters, and $P$ the number of pilot images, each labeled by all $R$ raters. The remaining $B - PR$ contours are single labels spread over new images. A first-order model of the error is
$$ E(P) \approx \frac{a}{P} + \frac{c}{B - PR}, $$
where $a$ measures how hard rater style is to estimate from a few shared images and $c$ measures how much the segmentation backbone benefits from more distinct images. Both terms have the usual one-over-sample-size form of estimation error. Minimizing gives
$$ P^{*} = \frac{B}{R + \sqrt{cR/a}}. $$
This is structurally the same balance wireless designers strike when choosing pilot length against the length of the coherence window. When raters drift, the pilots go stale. A pilot then only stays valid for about $T$ images per rater before it must be re-inserted, which raises the effective $a$. Our experiments will measure $a$ and $c$ empirically and check whether the predicted $P^*$ lands near the measured optimum. We treat this formula as a hypothesis to test, not a result.
Relation to Prior Work
The wireless papers supply the accounting logic, not the method. When network coding and dirty paper coding meet in a cooperative ad hoc network doubles high-signal-to-noise throughput with non-orthogonal relaying, but assumes perfect instantaneous channel knowledge and leaves its acquisition cost open. Its annotation analogue is full-panel labeling. Interference management with mismatched partial channel state information shows that nodes seeing only local channel gains can still beat simple time-sharing, provided they share global structural knowledge. Opportunistic Network Decoupling with Virtual Full-Duplex Operation in Multi-Source Interfering Relay Networks reaches optimal degrees of freedom from local measurements, but only with an enormous relay population. That result is a caution: schemes that avoid shared calibration entirely can demand impractical scale. In the medical setting, we read this as a reason to expect pure single-label protocols to need many more images to match a calibrated one.
The medical papers supply the models we reuse and the evidence for the gap. Disentangling Human Error from Ground Truth in Segmentation of Medical Images learns a segmentation network together with per-pixel rater confusion maps, regularized by a trace penalty. It already shows that a one-label-per-image regime is workable: on MSLSC it reports 76.48 Dice, against 83.29 with dense labels and 56.73 for STAPLE, a classical expectation-maximization label-fusion algorithm, per the arXiv version. It treats label sparsity as a given, however, and does not design which images are shared. Here rater parameters are identified only indirectly, through overlap in the data. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) represents each rater as a Gaussian in a latent space. The mean captures systematic bias and the variance captures inconsistency. It trains on dense multi-rater data. The Latent Doctor Model for Modeling Inter-Observer Variability encodes a rater from a support set of that rater's past annotations. This is the closest idea to a pilot set, but the support set is whatever history exists rather than a budgeted, shared, periodically refreshed design. Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle weights raters by downstream diagnostic value. We note it as a complementary way to value labels and do not adopt it, because it ties calibration to one classifier.
What is new here is the acquisition protocol and its budget law, not a new network. We keep the Disentangling and Pionono architectures fixed and change only which images get which labels.
Research Program
The first component is a pilot-aware training objective. In the Disentangling model, pilot images contribute all $R$ rater labels, so each rater's confusion maps are anchored on shared anatomy. Coverage images contribute one label, routed through the corresponding rater's confusion network. In Pionono, each rater's latent Gaussian is fit primarily from pilots and then shared across that rater's coverage images. A single weighting hyperparameter controls how strongly coverage images may update rater parameters. Setting it to zero gives pure pilot calibration, and letting it grow recovers the unmodified model.
The second component is the budget study. For each dataset we fix $B$ at several levels and sweep $P$ from zero up to the full-panel extreme. This traces the error curve $E(P)$, from which we fit $a$ and $c$ and compare the empirical optimum against $P^*$.
The third component is drift. None of the benchmarks record annotation order, so drift cannot be observed directly. We will therefore inject synthetic drift and state clearly that this is simulated. On MNIST-style synthetic segmentation, as in the Disentangling paper, and on LIDC-IDRI, we apply gradually increasing dilation or erosion to a rater's masks as a function of labeling position. We then compare one-time pilots against pilots re-inserted every $T$ images. Our motivation is the fatigue-driven variance documented for whole-body lesion tracing in qPSMA.
Baselines
The naive reference is majority vote or STAPLE applied to whatever labels the budget buys, training a single deterministic U-Net on the fused mask. On MSLSC, STAPLE under one label per image reaches 56.73 Dice and under dense labels 78.03 Dice, per Disentangling Human Error from the Ground Truth. This baseline tells us whether modeling raters matters at all under a budget.
The strongest competitor is the unmodified Disentangling model trained on the same budget spent as one label per image on as many distinct images as possible, with no pilots. It is the method that already handles sparse labels, and on MSLSC it reports 76.48 Dice in that regime. For probability-distribution metrics, the matching competitor is Pionono trained on the same budget. On full panels, Pionono reports Cohen's kappa of 0.758 on Gleason 2019 and 0.711 on the 25-rater breast cancer set, and the Latent Doctor Model reports a KL divergence of 0.461 on Gleason 2019. For LIDC-IDRI there is no reported result found on Lacuna for per-rater Dice with these methods, so we will run them ourselves.
A third baseline is full-panel labeling of $B/R$ images, the "perfect knowledge" extreme.
The key ablation keeps the pilot budget but spends it on random disjoint double labels instead of one shared set seen by all raters. This isolates whether the benefit comes from sharing the same images across all raters or merely from having more than one label per image.
For fairness, every arm uses the same backbone (a U-Net with a ResNet-34 encoder, as in Pionono), the same total contour count $B$, the same epochs, and the same hyperparameter search budget of 20 trials. Each arm is run with five seeds. Beating the strongest competitor means higher mean per-rater Dice and lower panel KL divergence, with non-overlapping 95% intervals, at at least two of three budget levels.
Why not just…?
Why not just give every image one label and let the disentanglement model infer rater styles? This is the real threat. Rater parameters may be identifiable from sparse overlap alone, and the MSLSC gap between 76.48 sparse and 83.29 dense Dice could reflect label count rather than missing calibration. The budget sweep settles it directly. The no-pilot protocol is the $P=0$ point on the same curve. If that point is the minimum, the pilot idea fails and we will report it. We expect pilots to help most when raters are many and each labels few images, as in the 25-rater breast set.
Why not just use active learning to request extra labels where the model is uncertain? Uncertainty-driven querying picks images one at a time and concentrates repeat labels on hard cases. That confounds rater style with image difficulty, which is the very separation rater models need. We include an uncertainty-sampling arm with a matched budget. If it matches pilots, the simpler adaptive scheme wins, and we would then test a hybrid in which pilots are chosen by uncertainty.
Why not just label fewer images with the full panel? That is the right-hand end of our sweep. The formula predicts it is optimal only when style estimation dominates the error, meaning $a$ is large relative to $cR$. The experiment reports where each dataset falls.
Why not just use STAPLE? Its collapse from 78.03 to 56.73 Dice under sparse labels on MSLSC indicates that fusion without a learned image model handles budgets poorly. It is still included as the naive reference.
Proposed Experiments
We use datasets where full panels exist, so any budgeted protocol can be simulated by subsampling and evaluated against held-out rater masks: LIDC-IDRI (up to four radiologists), Gleason 2019 (six pathologists), the 25-rater breast cancer set, MSLSC, and ISUP (20 pathologists, used for panel-entropy correlation, where the Latent Doctor Model reports a Spearman correlation up to 0.728).
We run three decisive checks. First, whether the error curve over $P$ has an interior minimum that beats both endpoints at equal $B$. Second, whether the fitted $P^*$ predicts that minimum within one sweep step across datasets with different $R$. Third, under simulated drift, whether periodic pilot refresh recovers per-rater Dice that one-time pilots lose. The shared-versus-disjoint ablation determines whether the effect is really calibration. A negative result on the first check, where $P=0$ wins, would itself be useful to practitioners planning labeling budgets, and we would report it as such.
References
- When network coding and dirty paper coding meet in a cooperative ad hoc network
- Interference management with mismatched partial channel state information
- Opportunistic Network Decoupling With Virtual Full-Duplex Operation in Multi-Source Interfering Relay Networks
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- Disentangling Human Error from the Ground Truth in Segmentation of Medical Images
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- qPSMA: Semiautomatic Software for Whole-Body Tumor Burden Assessment in Prostate Cancer Using 68Ga-PSMA11 PET/CT