When several radiologists outline the same scan, they often disagree about more than where the boundary lies. They also disagree about whether a lesion exists at all. On an ambiguous nodule, one reader draws a small contour and another draws nothing. Every multi-rater segmentation model we know of records the second reader's choice as an empty mask, a confident statement of "no tissue here," and trains on it next to the first reader's contour. We think this throws away the most informative part of the omission. A reader who skips a faint lesion is usually saying that whatever they perceived was too small or too faint to report. That is a censored measurement: a value known only to lie below some cutoff. Econometrics has handled this situation for decades with the Tobit model, which treats zeros as values that fell below a threshold rather than as genuine zeros.
We propose to model each rater's annotation of a lesion as a censored observation of a latent, rater-specific lesion extent. Each rater has a learned bias, meaning they systematically draw larger or smaller contours, and a learned detection threshold. When their perceived extent falls below that threshold, they draw nothing. When it falls above, they draw a contour of that size, and the shape comes from an image-conditioned decoder. The model makes a strong, falsifiable claim: omission and contour size are two expressions of one underlying quantity. We test that claim directly against the obvious alternative, in which "whether to annotate" and "how big to draw" are unrelated decisions.
Core Claim
Our central claim is that inter-observer variability on small or faint lesions is largely explained by a single latent extent per lesion and rater, observed through rater-specific censoring. Modelling it this way should improve three things over treating omissions as empty masks:
- It should give better-calibrated predictions of whether a given rater would annotate a lesion.
- It should give better-calibrated distributions of the volume a rater would draw, including the probability mass at zero.
- It should transfer to a new rater from a handful of their annotations, because the new rater is summarised by two interpretable numbers, a size bias and a detection threshold.
The Model
For lesion $i$ with image $x_i$ and rater $r$, let $L^*_{ir}$ be the latent log of the equivalent diameter that rater $r$ perceives:
$$L^*_{ir} = \mu(x_i) + b_r + \sigma(x_i)\,\varepsilon_{ir}, \qquad \varepsilon_{ir} \sim \mathcal{N}(0,1).$$
Here $\mu$ and $\sigma$ are outputs of a segmentation encoder, so the scatter of perceived size can vary with the image, which addresses the equal-variance fragility of classical Tobit. The term $b_r$ is the rater's size bias. The rater outlines a lesion only when $L^*_{ir} > \tau_r$, where $\tau_r$ is a learned per-rater threshold, and then draws a contour whose equivalent diameter is $\exp(L^*_{ir})$. The likelihood therefore has two cases:
- When the rater outlined the lesion, it is the Gaussian density at the observed log-diameter.
- When the rater omitted it, it is $\Phi\big((\tau_r - \mu - b_r)/\sigma\big)$, the probability that their perceived extent fell below their threshold, where $\Phi$ is the standard normal cumulative distribution.
These are exactly the two terms of the Tobit log-likelihood, except that the censoring point varies by rater. The threshold is identifiable because a rater's observed contours are truncated at $\tau_r$: the smallest sizes they ever draw pin down where their cutoff sits.
The shape of a drawn contour comes from a decoder that predicts a signed distance map, meaning the distance of each pixel from the boundary, signed by inside or outside. The decoder is conditioned on the image and on the sampled extent, and it is trained with a standard overlap loss on contours that were actually drawn.
To sample what rater $r$ would draw, we sample $L^*$. If it falls below $\tau_r$ we return an empty mask; otherwise we decode a shape at that size.
We also build a distribution-free variant that replaces the Gaussian with a grid of conditional quantiles of $L^*$. It is trained with the reweighting scheme of Censored Quantile Regression Neural Networks, which splits each censored point into a pseudo-point at the threshold and one far above it and learns all quantiles with a shared trunk. That paper worked only on one-dimensional survival data such as METABRIC and SUPPORT, where it reported better calibration in 10 of 12 instances and 12x to 34x training speedups. Our use differs in that each rater has their own censoring point and the regression target is an image-derived lesion extent.
Relation to Prior Work
The multi-rater segmentation literature supplies the backbone and the baselines.
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) gives each rater a Gaussian in an 8-dimensional latent space. The latent mean captures that rater's bias and the variance captures their inconsistency.
- The Latent Doctor Model for Modeling Inter-Observer Variability infers a doctor's style from a support set of that doctor's labels, but works on categorical grades rather than contours.
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts (MoSE) matches the distribution of predicted masks to the distribution of annotator masks on LIDC-IDRI using an optimal-transport loss.
All three treat an omitted annotation as one more mask, the empty one. Their rater representations are unstructured latent vectors, so nothing ties a rater's tendency to skip lesions to their tendency to draw small ones. We impose exactly that tie and test it.
Two further works shape the design in different ways.
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images models annotators as corrupting a single true mask through pixel-wise confusion matrices. That forces every disagreement to count as error. Our latent extent is rater-specific, so legitimate disagreement about size survives. The shared image-driven mean $\mu(x)$ still plays the role of a common anatomical signal.
- Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels shows that soft Dice loss pushes soft-label predictions toward 0 and 1. It reports expected calibration error on Prostate T1 dropping from 0.51 under majority voting to 0.20 with soft labels and its loss. This motivates taking boundary pile-up seriously.
The pile-up result also constrains us. It tempts the obvious cross-over idea of a two-limit Tobit loss on the pixel-wise agreement map, and we argue below that this is the wrong move.
Research Program
Data audit. We first measure how common omissions actually are. For each multi-rater dataset we count lesions annotated by some raters but not others, and we plot the size of drawn contours against how many raters drew them. If omissions are rare, or uncorrelated with contour size among those who did draw, the premise weakens and we report that.
Main models. We then train three models on shared encoder backbones:
- the parametric censored model;
- the censored-quantile variant;
- a two-part ("hurdle") model with the same backbone, in which a classifier predicts whether rater $r$ annotates and a separate regressor predicts log-diameter given annotation, with no shared latent variable.
Synthetic test. We simulate raters on LIDC-IDRI consensus masks with known biases and thresholds, by eroding or dilating contours and deleting them below a set size. This lets us check whether $b_r$ and $\tau_r$ are recovered.
Few-shot rater adaptation. Finally we freeze the network and fit only $b_r$ and $\tau_r$ for a held-out rater from 5 to 20 of their annotations. We compare this to Pionono-style adaptation of a full latent vector from the same examples.
Baselines
Naive reference. The do-nothing reference trains a single U-Net with the same backbone on majority-vote masks. A rater's omission probability is then taken as the model's probability that the lesion is absent, and its volume prediction is deterministic. Any method that fails to beat this on rater-level detection calibration has no claim.
Strongest learned competitor. This is MoSE on LIDC-IDRI, with Pionono as a second strong rater-aware baseline, since Pionono models individual raters explicitly. Both are run on our data splits with empty masks kept as targets, the same encoder family, the same number of training epochs, and the same hyperparameter-search budget as our model. Rater-level quantities are read off their samples: the fraction of samples that are empty, and the volume of non-empty samples.
The file reports no exact generalized energy distance (GED), the standard sample-diversity metric, for MoSE on LIDC-IDRI. MoSE says only that it "significantly reduced" GED relative to the Probabilistic U-Net and PHiSeg. So we will re-run it and state the figure we obtain. For Pionono, the file reports Cohen's kappa of 0.758 on Gleason 2019, but that is a grading task we do not use.
Beating these baselines means two things on held-out lesions:
- a lower Brier score for "will rater $r$ annotate this lesion";
- a lower continuous ranked probability score (CRPS) for the rater's drawn volume, with zero counted as a value, measuring how well a whole predicted distribution matches the outcome.
It also means doing so without a worse GED on the full mask distribution.
Ablations. Two ablations isolate the key ingredient. The hurdle model removes the coupling between detection and extent and keeps everything else. A fixed-threshold version shares one $\tau$ across raters, testing whether per-rater censoring points matter. For the QUBIQ tasks, the reference figures in the file are top teams' average Dice between 0.80 and 0.85 in QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge, and 0.5372 continuous Dice on Brain Growth from Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures. We use these only as checks that our segmentation quality is not degraded.
Why not just…?
Why not just keep empty masks as targets in a flexible latent-variable model like MoSE or Pionono? This is the real threat. A rich enough latent model can learn a zero mode and may match our GED. Our model's distinctive predictions are structural. It predicts that a rater's chance of omitting a lesion should fall smoothly as the lesion's predicted extent grows. It also predicts that a new rater should be described by two numbers. The few-shot adaptation and synthetic threshold-recovery experiments test exactly these predictions. If MoSE or Pionono match us there as well, our structural claim is not useful, and we will say so.
Why not just fit a two-part model, one classifier for detection and one regressor for size? This is cheaper and assumes less. It is the ablation that decides the scientific question. The Tobit coupling predicts something specific about lesions drawn by only one of four readers: their contours should be smaller, after accounting for image evidence, than the same readers' contours on lesions everyone drew. The hurdle model does not predict this pattern. If the data show no such dependence and the hurdle model scores as well, we will report the coupling hypothesis as false and recommend the hurdle model.
Why not just apply a two-limit Tobit loss to the pixel-wise agreement map, the obvious intersection? With $R$ raters, a pixel's agreement value is a count out of $R$. A binomial likelihood, which is cross-entropy with soft targets, is already the exact model for that count. A Tobit on the averaged value would be misspecified. It also says nothing about which rater omitted the lesion, or why. We include this pixel-level Tobit as a baseline to show the point empirically rather than assert it.
Why not just use STAPLE-style label fusion? It produces one consensus estimate with a performance level for each rater, but it has no notion of a size threshold. The Disentangling paper reports that learned annotator models beat STAPLE, including a 14.4% better confusion-matrix estimate on BraTS. So we do not expect STAPLE to compete on rater-level prediction. We include it only for completeness on the synthetic data.
Proposed Experiments
Primary benchmark. The primary benchmark is LIDC-IDRI, where each nodule has up to four radiologist readings. Partial omissions are, to our understanding, common, because readers outline only lesions they judge large enough; this is exactly the censoring regime. We use patient-level splits. As secondary tasks we take the QUBIQ lesion tasks (pancreatic lesion and brain tumor), provided the data audit finds per-rater empty masks there.
Metrics. We report:
- Brier score and calibration error for rater-level detection;
- CRPS for rater-level volume;
- coverage of the 80% intervals from the quantile variant;
- GED for the full mask distribution;
- Q-Dice on QUBIQ.
Decisive checks. Four results settle the proposal:
- Threshold recovery. On synthetic raters, recovered $\tau_r$ and $b_r$ should correlate with the true values, and the empty-mask baselines cannot report these quantities at all.
- Coupling test. On real LIDC-IDRI, the Tobit model must beat the hurdle ablation in held-out likelihood and in CRPS on lesions drawn by only some readers. If it does not, the core claim fails.
- Few-shot adaptation. With at most 20 annotations from a held-out reader, two-parameter adaptation should match or beat full-latent adaptation in detection Brier score.
- Robustness to the Gaussian assumption. The quantile variant should match the parametric model where the size residuals look Gaussian and improve on it where they are skewed, which we check with residual diagnostics.
References
- Censored Quantile Regression Neural Networks
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures