When several expert pathologists or radiologists grade the same image, they often disagree. Part of that disagreement comes from the image being genuinely ambiguous. Part comes from the experts setting different bars for calling a finding positive. A radiologist who fears missing a cancer will call more nodules "likely malignant" than a colleague with the same perception but a different tolerance for false alarms. Current methods for learning from many raters fold both sources into a single noisy label distribution. A model trained that way cannot say which part of its output reflects the image and which reflects one clinician's caution. It also cannot pick a decision rule that holds up when the deployment site's reference standard comes from raters it never saw.
We propose to split each rater's label into two pieces: a shared, image-dependent belief score and a rater-specific set of cutoffs that turn the score into an ordinal grade. We then choose the deployed decision rule by solving a small two-player game. In a robust game, the learner plays against an adversarial "Nature" that picks the worst case from a bounded set of possible worlds. Here Nature picks a plausible rater's cutoff profile, and the set Nature may choose from is estimated from the panel's observed disagreement rather than assumed. The claim is that this set over cutoffs is much tighter than a set over raw label distributions. Because of that, the resulting equilibrium decision rule should be less pessimistic than worst-rater training, should transfer better to held-out raters, and should expose systematic risk postures such as the "safety-first" bias documented on LIDC-IDRI.
Core Claim
Re-thinking and Re-labeling LIDC-IDRI relabeled LIDC-IDRI nodules by propagating biopsy labels with metric learning. It found that radiologist malignancy scores are skewed toward over-calling: more than half of score-4 nodules turned out benign, and a model trained on the original labels reached sensitivity 0.97 at specificity 0.23. That paper fixes the labels but does not model why raters shift.
Probabilistic rater models describe the shift without explaining it. Pionono represents each rater as a Gaussian in a learned latent space. The Latent Doctor Model encodes rater style with a conditional variational autoencoder. Both reproduce each rater's label distribution well. Neither says which part of the latent code is a threshold and which is perception, and neither yields one decision rule with a worst-case guarantee over raters.
Our claim is that much of inter-observer variability on ordinal grading tasks is explained by rater-specific cutoffs on a shared score. Turning those cutoffs into the uncertainty set of a robust game gives a deployed operating point with three properties. It is guaranteed against any rater in the calibrated set. It is less conservative than worst-case training over raw labels. Its implied cutoffs on LIDC recover the safety-first bias without using biopsy labels during fitting.
Model and Game
Rater $r$ assigns grade $k$ to image $x$ when $s_\theta(x)+\varepsilon$ falls in the interval $(c_{r,k-1}, c_{r,k}]$. Here $s_\theta$ is a network producing the shared belief score, $\varepsilon$ is logistic noise capturing intra-rater inconsistency, and $c_r=(c_{r,1}<\dots<c_{r,K-1})$ are rater $r$'s cutoffs. This is an ordinal regression with rater-specific cutpoints, the classical graded-response model from psychometrics. We make no claim that the measurement model itself is new.
What is new is the next step. We fit $\theta$ and all $c_r$ jointly, bootstrap over images to get the sampling spread of the cutoffs, and define the set Nature may choose from, $\mathcal{C}$. The set is the convex hull of the estimated rater cutoff vectors, enlarged by the bootstrap radius. Taking the hull lets Nature also choose unseen raters who interpolate between observed ones.
The learner chooses deployed cutoffs $\delta$ on the score, and Nature chooses $c\in\mathcal{C}$:
$$\min_{\delta}\ \max_{c\in\mathcal{C}}\ \mathbb{E}_x\big[\ell(\hat y_\delta(x), y_c(x))\big] - \min_{\delta'}\mathbb{E}_x\big[\ell(\hat y_{\delta'}(x), y_c(x))\big],$$
In this objective, $\hat y_\delta$ is the grade the deployed rule assigns, $y_c$ is the grade a rater with cutoffs $c$ would assign, and $\ell$ is an asymmetric ordinal loss. The objective is worst-case regret relative to the best rule for that particular rater. Mixing over $\delta$ and $\mathcal{C}$ convex gives a saddle point by Sion's minimax theorem, the same tool Distributionally Robust Game Theory uses to reduce robust equilibria to finite systems. That paper's ambiguity sets are synthetic. Ours come from data, which addresses the calibration gap noted for robust games in Breaking the Curse of Multiagency.
We also report a variant that replaces the max with a conditional value-at-risk (the average over the worst fraction of raters) to tune pessimism. Recovering rater cutoffs from observed labels plays the role that inverse variational-inequality fitting of aggregator weights plays in Inverse learning of black-box aggregator. We reuse the idea of learning the game's structure from observed behavior before solving its robust counterpart, not their algorithm.
The score and cutoffs are identified only up to a monotone transform. As a result, "bias" means bias relative to the panel unless an external anchor exists. LIDC has such an anchor in the SCH-LND biopsy set; Gleason and ISUP do not. We keep these two cases separate in evaluation.
Research Program
The program has three parts. First, we test whether the cutoff decomposition fits as well as free-form rater models. We compare held-out-image likelihood of each rater's labels under three models: ours, Pionono, and a cutoff model augmented with a low-rank per-rater perception term. If the perception term buys a lot, the decomposition is too crude and the method needs that extension.
Second, we solve the robust game and compare its decision rule against alternatives under rater shift. The key stress test is leave-raters-out: fit on some raters, evaluate against raters never seen.
Third, on LIDC-IDRI we fit cutoffs to the four-radiologist scores without biopsy information. We then check whether the inferred cutoffs sit systematically low relative to cutoffs fitted on SCH-LND biopsy truth. Finally we check whether the robust rule, when evaluated against biopsy labels, moves off the 0.97/0.23 operating point.
Baselines
The naive reference is majority vote with cross-entropy, plus soft-label training on the empirical vote distribution. The Latent Doctor paper already shows majority vote leaves models overconfident on borderline cases. Beating it is required but uninformative.
The strongest existing rater models are Pionono, with reported unweighted Cohen's kappa 0.758 on Gleason 2019 and 0.711 on the breast cancer segmentation set, and the Latent Doctor Model, with reported KL divergence 0.461 on Gleason 2019 for its conditional variant. To deploy them against an unknown rater, we give each the same robust selection step our method uses. That means sampling rater latents from their learned prior or support-set encoder and choosing the decision rule that minimizes worst-sampled-rater regret. This isolates the uncertainty-set geometry from the selection procedure. All methods share the same backbone and image splits, and each gets the same hyperparameter-search budget, tuned on validation raters.
The robust-optimization competitor is group distributionally robust training over raters: minimize the worst single-rater loss over raw labels. It shares our game structure but places the uncertainty on label distributions instead of cutoffs, so it is the ablation for our key ingredient.
On LIDC the label-repair baseline is Zhang et al.'s metric-learning relabeling, with reported sensitivity 0.67 and specificity 0.66 against biopsy-confirmed labels. That method uses biopsy anchors during training and ours does not, so matching it would be a strong result and falling somewhat short would still be informative.
We count as beating the field two outcomes. The first is lower worst-held-out-rater regret than both group DRO and the robustified Pionono/Latent Doctor rules, with no more than a small loss in average-rater kappa. The second, on LIDC, is a higher specificity at matched sensitivity than the original-label baseline. We also report the cutoff-free ablation, which uses the same game but takes Nature's set as the hull of per-rater label distributions.
Why not just…?
Why not just train Pionono or the Latent Doctor Model and condition on a "typical" rater? These models already reproduce individual raters well, and their latent codes may implicitly contain thresholds. This is the most serious threat. Two experiments settle it. We regress their per-rater latent means on our fitted cutoffs; a near-perfect fit means the decomposition adds interpretability but little else. We also compare held-out-rater regret after giving them the identical robust selection step. If they match us there, the contribution shrinks to an interpretable reparameterization, and we will say so.
Why not just do worst-rater robust optimization (group DRO) without a game? Our game is also learner-versus-Nature, so the honest difference is only where the uncertainty lives. Distributionally Robust Game Theory argues that unstructured worst-case sets are overly pessimistic. Our prediction is concrete: group DRO's guarantee covers only observed raters, and its average-rater performance drops more than ours. The cutoff-free ablation tests this directly. If it performs as well, the idea fails.
Why not just fuse labels with majority vote or STAPLE, the classical method that jointly estimates rater accuracy and a consensus label? The LIDC results show consensus inherits the panel's shared bias, giving specificity 0.23. The Disentangling Human Error paper reports STAPLE trailing a deep confusion-matrix model by 14.4% in confusion-matrix estimation on BraTS. Both baselines are included, but neither gives a worst-case statement about unseen raters.
Proposed Experiments
On Gleason 2019 (6 pathologists, 4 classes) and ISUP (20 pathologists, 6 classes), we run leave-raters-out cross-validation: hold out 2 and 5 raters respectively, rotating across folds. We report worst and mean held-out-rater kappa and regret, KL divergence to the held-out vote distribution, and a check that the ISUP held-out raters' fitted cutoffs fall inside or near $\mathcal{C}$. The 3k Buds dataset (9 pathologists, 3 classes) serves as a further ordinal test; no reported result was found on Lacuna for it. The bc segmentation set's 25 raters allow a pixel-level version that treats per-pixel class grades ordinally where the classes admit an order.
On LIDC-IDRI we fit on radiologist scores only, then evaluate sensitivity and specificity against SCH-LND-propagated biopsy labels. The comparison is against the 0.97/0.23 and 0.67/0.66 operating points reported by Zhang et al.
The decisive checks are three. First, the cutoff model's held-out likelihood must be within a small margin of Pionono's; otherwise the decomposition is wrong for these data. Second, our worst-held-out-rater regret must beat the cutoff-free ablation; otherwise the uncertainty geometry is not doing the work. Third, on LIDC, the radiologists' inferred cutoffs must sit below the biopsy-fitted cutoffs for the malignant grades, which would confirm that the method recovers the safety-first posture without supervision. A failure of the third check while the first two pass would mean we have a better robust rule but not a model of risk posture, and we would report it that way.
References
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Distributionally Robust Game Theory
- Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning
- Inverse Learning of Black-box Aggregator for Robust Nash Equilibrium