Rule-Out Guarantees When the Graders Disagree

AI triage tools in medicine are increasingly asked to make one promise. When the model says "negative," the patient really is negative, with a stated error rate and stated confidence. That promise is a guarantee on negative predictive value (NPV), the fraction of model-negative cases that are truly negative. It can be made without distributional assumptions by calibrating a decision threshold on held-out data. The weak point is the word "truly." Many conditions are graded on ordered severity scales such as normal, mild, moderate, severe. On these scales experts disagree constantly, and the disagreement piles up at exactly the normal-versus-mild boundary where a rule-out decision is made. Three laryngologists agreed unanimously on only 49% of vocal-fold images in Classifying Phonotrauma Severity from Vocal Fold Images with Soft Ordinal Regression. A threshold calibrated against one grader's labels therefore guarantees NPV with respect to that grader, not with respect to the clinical panel the hospital actually cares about.

We propose to make rule-out guarantees hold with respect to a panel of experts while paying for panel labels on only a small subset of cases. The key observation is that grading noise on ordered scales is not arbitrary. Disagreements are almost always between neighbouring grades: 95% one-neighbour accuracy versus 65% exact accuracy in Automatic hip osteoarthritis grading with uncertainty estimation from computed tomography using digitally-reconstructed radiographs. As a result, the noise that matters for a rule-out decision reduces to a couple of boundary-flip rates. These rates can be estimated from a few dozen multiply-graded cases and folded into the threshold calibration with honest confidence intervals.

Core Claim

Consider a single-rater calibration set and a small multi-rater audit subset drawn from the region the model would rule out. A two-parameter noise correction is applied only at the lowest severity boundary, and its estimation error is propagated into the calibration procedure. With these pieces, one can certify a panel-referenced NPV, and a separate bound on missed severe cases, for an ordinal grading model. Naive calibration on single-rater labels is either invalid or wasteful: invalid when the rater under-calls mild disease, wasteful when the rater over-calls it. The ordinal structure does two jobs. It justifies restricting the noise model to adjacent-grade flips. It also gives a severity-weighted miss criterion, under which missing a severe case is constrained far more tightly than missing a borderline-mild one.

Relation to Prior Work

Two lines of work supply the machinery, and both assume the labels are correct.

Conformal Triage for Medical Imaging AI Deployment uses Learn-then-Test calibration (a multiple-testing procedure that certifies which thresholds meet a risk target) to sort binary head-CT predictions into guaranteed-NPV, guaranteed-PPV, and abstain groups. We reuse this procedure directly. It is binary, though, and treats calibration labels as fixed. Its own noisy NLP-labelled cohort is handled as distribution shift, not as a label-error model.

Conformal Risk Control for Ordinal Classification and Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets give ordinal guarantees, but as contiguous prediction sets controlled on average over the population. Both treat single-source labels as ground truth. The first also reports a "centripetal" pull of sets toward middle grades, which is a direct obstacle to confidently ruling out grade 0.

The noise literature motivates our approach but does not provide a guarantee. Classifying Phonotrauma Severity from Vocal Fold Images with Soft Ordinal Regression trains cumulative "is severity above k?" classifiers on vote-fraction targets. It reaches QWK 0.85 with normal-class recall 0.98, but that recall is an empirical by-product, not a certified rule-out, and the method needs every image multiply graded. We use its training recipe as our scoring model.

LiNC learns per-sample trust from flat categorical disagreement. It warns that trust weighting reinforces errors that raters share. That warning constrains our design: correction must be anchored to an external panel subset, not inferred from the model's own agreement with labels.

Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment shows that three-rater consensus halves performance variance and that additional raters add little. This supports defining the reference panel as three graders and keeps the audit cost bounded.

Research Program

The scoring model is a soft ordinal network trained on whatever labels exist. Its rule-out score is $s(x)=\hat P(Y\ge 1\mid x)$, the predicted probability of any disease, and cases with $s(x)\le\lambda$ are ruled out.

The target is defined against a panel reference grade $Y^{p}$. This is the panel median for the main analysis, with a stricter "any panel member grades ≥1" variant reported alongside. We seek the largest $\lambda$ such that, with probability at least $1-\delta$ over calibration,

$$ P(Y^{p}\ge 1 \mid s(X)\le\lambda)\le\epsilon_1 \quad\text{and}\quad P(Y^{p}\ge 2 \mid s(X)\le\lambda)\le\epsilon_2, $$

where $\epsilon_1$ is the tolerated rate of missed disease of any grade, $\epsilon_2\ll\epsilon_1$ is the tolerated rate of missed moderate-or-worse disease, and $\delta$ is the allowed probability that calibration itself fails.

The calibration set carries only single-rater grades $\tilde Y$. Within the candidate rule-out region we write $a=P(\tilde Y\ge1\mid Y^p=0)$ for the rater's over-call rate and $b=P(\tilde Y=0\mid Y^p\ge1)$ for the rater's under-call rate. The observed miss rate $\tilde p$ then relates to the panel miss rate $p$ by the standard misclassification identity

$$ p=\frac{\tilde p-a}{1-a-b}. $$

We estimate $a$ and $b$ on the audit subset, restricted to cases with low scores. This restriction matters because noise near the decision boundary differs from noise across the whole dataset. Upper confidence bounds for $a$, $b$, and $\tilde p$ are combined by a union bound. The resulting conservative estimate of $p$ is fed to Learn-then-Test, which scans thresholds from most to least conservative.

The second constraint gets the same treatment at the 1/2 boundary. If the audit data confirm that flips skipping a grade, such as 0 versus 2, are rare, this correction is small. That is the ordinal assumption, stated so it can be tested: we report the audit-estimated non-adjacent flip rate and fall back to a full confusion correction when that rate is non-negligible.

We will also derive the audit size needed for the correction to cost less than a fixed fraction of rule-out yield. We expect the interval width on $p$ to scale as $1/((1-a-b)\sqrt{n_{\text{audit}}})$, the usual binomial rate inflated by the correction denominator. This suggests audits of 50–150 low-score cases. We will check that estimate empirically.

Baselines

The do-nothing reference rules nobody out and sends every case to a specialist. It defines zero yield and zero misses. Every method is scored by the fraction of truly negative cases it removes from the queue, at a given certified error.

The naive guarantee runs Learn-then-Test, exactly as in Conformal Triage for Medical Imaging AI Deployment, on the single-rater calibration labels collapsed to negative versus positive. It uses the same scoring model, the same calibration cases, and the same $\epsilon_1$ and $\delta$. This is the strongest existing competitor because it already gives a formal NPV certificate. The only thing it lacks is a model of rater error. That paper reports false positives falling from 233 (45%) to 8 (5%) at a 14% abstention rate on head CT, which is not one of our benchmarks. For NPV on our ordinal benchmarks, no reported result was found on Lacuna.

We beat this baseline if, against held-out panel labels, it violates the NPV target noticeably more often than $\delta$ under under-calling raters while ours stays at or below $\delta$. We also beat it if, under over-calling raters, it rules out significantly fewer negatives than ours at equal validity.

The oracle runs the same Learn-then-Test with full panel labels on the entire calibration set. It upper-bounds what we can achieve. Our goal is to recover most of its yield while paying for panel labels on only a small audit fraction.

The ordinal-set baseline applies ordinal conformal risk control from Conformal Risk Control for Ordinal Classification and rules a case out when its prediction set is exactly {0}. It is calibrated to the same budget. Its reported Diabetic Retinopathy result is empirical risk almost exactly 0.08 at nominal 0.08, but that figure concerns set risk, not NPV.

The scoring model's own reference is soft ordinal regression Classifying Phonotrauma Severity from Vocal Fold Images with Soft Ordinal Regression, with its argmax-normal rule. The paper reports QWK 0.85, ECE 0.10, and normal recall 0.98 on the phonotrauma data, without any certificate.

Two ablations isolate the key ingredient. The first estimates $a$ and $b$ over the whole audit set instead of only the low-score region. The second replaces the ordinal scoring model with a binary classifier trained on collapsed soft labels.

Why not just…?

Why not just calibrate on majority-vote labels? If every calibration case has three or more grades, plain Learn-then-Test on majority labels is already valid for a majority-referenced NPV, and our correction adds nothing. That configuration is our oracle. The proposal only matters when panel labels are too expensive for the full calibration set, which Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment and practical cost both suggest is the common case. The decisive measurement is yield versus audit size: how close we get to the oracle at, say, a 10% audit.

Why not just collapse to binary and use a conservative threshold margin? A hand-picked safety margin can be valid by luck, but it carries no certificate and wastes yield whenever the rater over-calls. It also cannot separately bound severe misses. We include a "margin tuned on validation" variant and report how often it fails across rater-noise conditions. The binary-scorer ablation additionally tests whether ordinal training improves the rule-out score itself. If it does not, we will report that the ordinal contribution lies in the severity-weighted constraint and the noise model, not in the representation.

Why not just use ordinal conformal sets and rule out when the set is {0}? This is a genuine competitor, because it already uses order. Its guarantee is marginal coverage over all patients, though, not error among ruled-out patients. Any case whose set is {0} still has a population-averaged miss probability rather than a conditional one. It also trusts labels, and the centripetal effect reported in Conformal Risk Control for Ordinal Classification should shrink the set of cases receiving {0}. The head-to-head comparison at matched budgets settles which effect dominates.

Why not just denoise labels with LiNC first? Self-consistency trust cannot detect a rater who systematically under-calls mild disease, a limitation LiNC acknowledges. Its output also carries no statistical guarantee. We will run LiNC-cleaned labels through naive Learn-then-Test and measure panel-referenced violations.

Proposed Experiments

The controlled study uses the Diabetic Retinopathy five-grade benchmark and the lumbar spine stenosis dataset from Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets. Existing labels serve as the panel reference. We then simulate single raters with known adjacent-flip rates in three directions (under-calling, over-calling, symmetric), plus a fraction of non-adjacent flips that stress the ordinal assumption. Over many random calibration/test splits we record:

  • the panel-referenced violation frequency of each NPV and severe-miss constraint, compared with $\delta$;
  • rule-out yield;
  • yield as a function of audit size.

The real-disagreement study uses the MGH phonotrauma images with three surgeon grades. It holds out one surgeon as the "single rater" and the other two plus that one as the panel, rotating across surgeons. With about 150 multiply rated images, the certificates here will be loose. We treat this dataset as a test of whether real rater flips are concentrated at adjacent grades and within the low-score region, not as a headline yield number.

If access permits, the gastric dysplasia whole-slide images, which have a 12-pathologist observer study over negative, low-grade, and high-grade dysplasia (Deep learning for automatic diagnosis of gastric dysplasia using whole-slide histopathology images in endoscopic specimens), provide a larger real panel. There, ruling out "negative for dysplasia" is exactly the clinical question.

The decisive checks are four. First, naive single-rater Learn-then-Test must visibly break its guarantee under under-calling raters; if it does not, the problem is smaller than claimed. Second, our corrected procedure must hold the guarantee at $\delta$ while recovering most of the oracle's yield at a small audit size. Third, the low-score-region noise estimate must outperform the global estimate. Fourth, as the injected rate of non-adjacent flips rises, we expect the two-parameter correction to fail detectably and the fallback to restore validity. This would show that the ordinal assumption is doing real work and is checked, not merely assumed.