When several experts label the same scan and disagree, the usual move is to collapse their votes into one "gold" label, train a scoring model, and then choose a decision cutoff that maximizes sensitivity plus specificity on that label. This note argues that the disagreement often has a simple structure that this pipeline throws away. Experts largely agree on which cases look worse than others, but each one draws the line at a different place. A cautious radiologist calls a borderline nodule malignant, while a less cautious one does not. If we model each rater as applying a personal cutoff to a shared, learned risk score, three things follow. First, the ranking of cases can be learned from every rater's labels at once. Second, each rater's implicit operating point becomes an explicit, inspectable number. Third, choosing the clinically correct cutoff collapses to estimating a single scalar, which a handful of outcome-confirmed cases can pin down.
The stakes are concrete. On LIDC-IDRI, a lung-nodule CT benchmark labeled by up to four radiologists, a model trained on consensus malignancy ratings reached sensitivity 0.97 but specificity 0.23. The cause is the radiologists' "safety-first" habit of upgrading uncertain nodules. Fixing this required relabeling against 180 biopsy-proven nodules (Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction). We ask whether a rater-cutoff model can reach the same corrected operating point with roughly a tenth of those anchors. We also ask exactly how much can be said with no anchors at all.
Core Claim
Let $s_\theta(x)$ be a learned scalar risk score for case $x$. Rater $k$ gives ordinal rating $r_{ik} \ge m$ on case $i$ with probability
$$P(r_{ik} \ge m \mid x_i) = \Phi\!\left(\frac{s_\theta(x_i) - c_{k,m}}{\sigma_k}\right),$$
where $\Phi$ is the standard normal cumulative distribution function. The term $c_{k,m}$ is rater $k$'s cutoff for category $m$, and $\sigma_k$ is rater $k$'s noise scale, so a larger $\sigma_k$ means a less consistent rater. True disease status is modeled the same way, as $D=1$ when $s_\theta(x)$ plus noise exceeds an unknown clinical cutoff $c^*$.
This is a deliberately low-dimensional model of disagreement. It is a cumulative-probit ordinal model with rater-specific thresholds and a neural score. It produces exactly the instance-dependent noise documented for human labels: errors pile up for cases whose score lies near a rater's cutoff.
The claim has three parts. The first is an identifiability statement. From multi-rater labels alone, the case ranking under $s_\theta$ and the relative positions of all rater cutoffs are identified. The absolute location of $c^*$ is not identified, because a panel that is uniformly biased looks the same as an unbiased panel on a shifted axis. The second is a negative result for the standard pipeline. When rater cutoffs differ from $c^*$, maximizing the Youden index against majority-vote or single-rater labels converges to a point between the rater cutoffs rather than to $c^*$. The assumption that rescues the single-noisy-reference case, conditional independence of label errors from the biomarker, fails by construction here. The third is a positive result. Because only the scalar $c^*$ remains unknown once $s_\theta$ is fixed, it can be estimated from $n_a$ anchor cases with error shrinking like $1/\sqrt{n_a}$, the standard rate for one parameter. A cutoff tuned directly on anchors must instead also learn the ranking.
What This Builds On and Departs From
The threshold-optimization side supplies the target and the smoothing technique. Biomarker combination based on the Youden index with and without gold standard combines biomarkers linearly by maximizing a smoothed AUC, then picks a Youden cutoff. It proves that a single imperfect reference can be treated as gold when its errors are conditionally independent of the markers. We reuse its normal-CDF smoothing, but our setting violates its assumption on purpose. Our analysis predicts where its guarantee breaks.
Biomarkers selection and combination based on the weighted Youden index adds an asymmetry weight $\pi$ between sensitivity and specificity, and it names noisy multi-observer references as open work. We adopt its weighted index so that $c^*$ can be chosen for a screening or a confirmatory use.
The label-noise side motivates the model and constrains it. Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations shows that human errors are instance-dependent and concentrate on ambiguous cases. That pattern is what a cutoff model predicts. The Impact of an Inter-rater Bias on Neural Network Training shows that networks trained on individual raters amplify those raters' systematic biases rather than averaging them out. This is a warning that single-rater training will bake a rater's cutoff into the model.
Richer multi-rater models instead encode rater style in high-dimensional spaces. The Latent Doctor Model for Modeling Inter-Observer Variability uses a conditional variational autoencoder over doctor styles, and Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks uses rater-specific decoders. Both capture more than threshold shifts, but neither isolates an operating point. We trade expressiveness for an explicit scalar per rater. We also test whether that trade loses accuracy.
Leveraging Inter-Rater Agreement for Classification in the Presence of Noisy Labels recovers a class-level noise matrix from agreement statistics without anchors. Its noise is class-level rather than instance-level, it assumes homogeneous symmetric raters, and it evaluates by argmax accuracy. Our model is instance-dependent and rater-asymmetric by design, and its output is a cutoff.
Difficulty-aware Glaucoma Classification with Multi-Rater Consensus Modeling mimics aggressive and conservative raters with oversampled network branches but never states their cutoffs. Our model places those stances as numbers on one axis.
Research Program
The work has three parts.
The first is analysis. Under the probit model we derive the limit of the empirical Youden cutoff computed against majority-vote labels and against each single rater. This gives a closed-form bias as a function of the gaps $c_{k,m} - c^*$ and the scales $\sigma_k$. We also state precisely which quantities are identified from rater labels alone.
The second is estimation. We train $s_\theta$, a ResNet-style image encoder or a linear marker combination for tabular data, jointly with $\{c_{k,m}, \sigma_k\}$ by maximum likelihood over all individual ratings. Absent raters are simply missing terms. We then estimate $c^*$ from $n_a$ outcome-confirmed anchors by maximizing a smoothed weighted Youden index of $s_\theta$ on the anchors, with $\pi$ set by the clinical use case.
The third is the anchor-free regime. Here we report the fitted rater cutoffs as an explicit interval on the risk axis. We report the cutoff implied by any named policy, such as "most specific rater" or "median rater." We present these as policy choices, not as estimates of $c^*$, because our own identifiability result says no anchor-free estimate of $c^*$ exists.
Baselines
The naive reference point is majority vote followed by Youden. We average or majority-vote the ratings into a binary label, train the same encoder with cross-entropy, and pick the Youden-optimal cutoff on a validation split labeled the same way. This is what most pipelines do, and our analysis predicts it lands between rater cutoffs. On LIDC-IDRI the closest reported analogue is the consensus-trained model in Re-thinking and Re-labeling LIDC-IDRI, at sensitivity 0.97 and specificity 0.23. That is a Youden index of 0.20, computed from the reported numbers.
The strongest existing method on the decisive question is that paper's anchor relabeling. A Siamese network trained on all 180 SCH-LND biopsy nodules propagates labels to LIDC-IDRI, and the reported result is sensitivity 0.67 and specificity 0.66, a Youden index of 0.33 computed from those figures. We run it with the same encoder backbone, the same LIDC split, and the same pathology-confirmed evaluation set. We beat it if our model, given a random subset of anchors, matches its Youden index with at most 20 anchors. Beating it outright at 180 anchors would be a bonus, not the claim.
The second strong competitor is agreement-based noise correction (Leveraging Inter-Rater Agreement). We estimate the noise matrix from the ratings, train with its posterior soft targets, and take the Youden cutoff on anchors. On CIFAR-10N it reports 79.4% accuracy against 74.0% for majority vote with ResNet-34. No reported result was found on Lacuna for this method on LIDC-IDRI or on any Youden metric.
We also run the Latent Doctor Model's predictive mean as a score, followed by anchor Youden. This tests whether a richer rater model beats ours once both receive the same anchors.
The ablation that isolates our key ingredient keeps everything identical but forces a single shared cutoff for all raters, $c_{k,m} = c_m$. If rater-specific cutoffs do not improve anchor efficiency or held-out-rater likelihood over this, the central idea fails.
For every method, the anchor set, encoder, training budget, and number of hyperparameter trials are matched. Anchor subsets are resampled 50 times, so all comparisons report means with intervals.
Why not just…?
Why not just tune the cutoff on the anchors directly? This is the most serious threat. If 20 biopsy cases suffice to pick a cutoff for a majority-vote model, the rater model adds nothing. The answer has to be empirical. Our anchor-efficiency curve compares anchor-only cutoff tuning on the majority-vote score against anchor-only tuning on $s_\theta$. Both choose one scalar, so any gain must come from a better ranking. There is a second, subtler threat: the gain might come from the encoder being trained on individual ratings rather than from the cutoff structure. The shared-cutoff ablation separates those two sources. If the curves coincide, we will report that the practical advice is "train on individual ratings, tune on anchors," and drop the stronger claim.
Why not just use a latent-class or Dawid-Skene style correction and skip anchors? Class-level confusion models, including Leveraging Inter-Rater Agreement, treat each rater's error rate as fixed per class. Under a cutoff model, however, error depends on how close a case's score is to the rater's line. The anchor-free versions also share our non-identifiability: a panel-wide safety-first shift is invisible to agreement statistics. The LIDC-IDRI comparison tests directly whether the corrected noise matrix moves the cutoff toward the biopsy-supported one.
Why not just model raters with a full latent-style network? The Latent Doctor Model can represent disagreements that are not threshold shifts, such as raters attending to different features. Where such disagreements dominate, it should fit held-out raters better than our model. We test this on Gleason2019 and ISUP, where many pathologists grade each case. If the gap in held-out-rater likelihood is large, the cutoff model is too crude for that domain. We would then restrict our claim to domains with an ordinal severity scale, such as nodule malignancy or cup-to-disc ratio.
Proposed Experiments
The first experiment is semi-synthetic and has known ground truth. We take binary tasks from MedMNISTv2 and simulate panels of raters with chosen cutoffs and noise scales on a hidden true score, using either a clean label or a held-out model's logit as that score. This lets us check the predicted Youden bias of majority vote against its closed form. It also traces anchor-efficiency curves as the spread of rater cutoffs grows. The prediction is that the advantage of rater-specific cutoffs grows with the spread and disappears when all raters share $c^*$.
The second experiment is decisive. On LIDC-IDRI, we fit the model to the individual 1–5 radiologist ratings, estimate $c^*$ from $n_a \in \{5, 10, 20, 45, 90, 180\}$ SCH-LND anchors, and evaluate sensitivity, specificity, and Youden index on held-out pathology-confirmed nodules. The baselines above serve as comparisons. We also check whether the fitted radiologist cutoffs sit below the anchor-estimated $c^*$, which is what the safety-first account predicts. A null here, with fitted cutoffs straddling $c^*$, would contradict the relabeling paper's interpretation and is worth reporting either way.
The SCH-LND anchors come from a different institution than LIDC-IDRI. To test whether the scalar $c^*$ transfers or is distorted by scanner shift, we compare it with a cutoff tuned on a held-out SCH-LND portion.
The third experiment tests whether the model describes real panels. On Gleason2019 (six pathologists) and ISUP (twenty pathologists), we hold out each rater in turn and predict that rater's grades from the fitted cutoffs. We compare this with the Latent Doctor Model's reported KL divergence of 0.461 on Gleason2019, using the same metric on the same split.
For REFUGE, where multiple ophthalmologists grade glaucoma, we report weighted Youden at a screening-oriented $\pi$. We compare against the 97.94% AUC reported by Difficulty-aware Glaucoma Classification. That figure measures ranking, not operating point, so it serves as a sanity check that our score does not rank worse while our cutoffs are more interpretable.
References
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Biomarker Combination Based on the Youden Index With and Without Gold Standard
- Biomarkers selection and combination based on the weighted Youden index
- Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
- The Impact of an Inter-rater Bias on Neural Network Training
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- Leveraging Inter-Rater Agreement for Classification in the Presence of Noisy Labels
- Difficulty-Aware Glaucoma Classification with Multi-rater Consensus Modeling