Whose Contour Sets the Cutoff? Rater-Marginal Thresholds for Image-Derived Biomarkers

Many clinical decisions come down to a number measured from an image and compared against a fixed cutoff. A common example is the vertical cup-to-disc ratio (vCDR) in glaucoma: the height of the pale central "cup" of the optic nerve head divided by the height of the whole disc, both traced on a fundus photograph. The number depends on where a person, or a model trained to imitate people, draws the boundary, and expert boundaries disagree. The cutoff is itself usually derived from contours drawn by one group of experts. So two separate things can move when the tracer changes. A patient's value can land on the other side of the cutoff, and the cutoff itself can drift, much as cohort-median cutoffs drift between hospitals in circulating-biomarker research. This note proposes to measure both effects on public multi-rater data. It then proposes deriving cutoffs from the full distribution of plausible tracings rather than from one consensus mask, and replacing the point measurement at deployment with a calibrated probability that the patient's value lies above the cutoff.

In short, the proposal treats "which rater drew this?" the way biomarker standardization treats "which cohort was this cutoff derived in?" It averages over raters when choosing the cutoff, and it reports a per-patient crossing probability with an explicit indeterminate zone. The test is whether this lowers the worst-case loss of sensitivity and specificity when the test-time tracer differs from the one used to set the cutoff, compared with a strong deterministic segmenter and a consensus-derived cutoff.

Motivation and Position Relative to Prior Work

Biomarker standardization research supplies the motivation. Feng et al. define "high" neutrophil-to-lymphocyte ratio by their cohort median of 3.24, so the rule is tied to that cohort's composition. Wei et al. pool 12 gastric cancer studies whose albumin-to-globulin cutoffs range from 1.14 to 1.93, with between-study heterogeneity of $I^2 = 76.8\%$. Both papers show that a cutoff estimated inside one population does not transfer. Neither paper concerns images. The analogous source of drift for image-derived markers is the human who traces the boundary, and nobody has measured it in that framing.

Segmentation-variability research shows that the drift source is real but stops short of the threshold. Chen et al. built a thyroid CT radiomic nomogram (validation AUC 0.812) and name manual contour instability as the main weakness, but they did not measure it. Three probabilistic models each learn a distribution over expert outputs and evaluate it with segmentation- or label-level metrics. Pionono gives each rater a Gaussian in a latent space to separate systematic bias from inconsistency, reaching κ = 0.758 on Gleason 2019. The Latent Doctor Model uses a conditional variational auto-encoder over pathologist labels. The variational inference model of Liu et al. reaches continuous Dice 0.5372 on QUBIQ brain growth. None of the three asks whether sampled raters would change a downstream continuous measurement's position relative to a clinical cutoff. These papers supply candidate methods for one component of the proposal, the per-patient crossing probability. They do not supply the threshold analysis.

Two papers are the closest bridges. DiFF fuses multi-rater masks with weights learned from a downstream diagnosis network, gaining up to 2.73% AUC on REFUGE-2. It optimizes a binary classifier rather than a continuous biomarker, it ties its masks to a particular classifier backbone, and its authors list the link to clinical biomarkers as unexplored. The proposal keeps the biomarker and its cutoff explicit, which is the part DiFF gives up. REFUGE pairs seven-specialist delineations with vCDR and glaucoma labels. It reports a vCDR-only AUC of 0.9471, below the best end-to-end classifier at 0.9885, and finds that changing the camera degrades performance. However, it evaluates deterministic models against a fused consensus and never studies rater-dependent cutoffs.

One paper constrains the design. Zhang et al. show that LIDC-IDRI radiologist consensus carries a "safety-first" malignancy bias: models trained on it reach sensitivity 0.97 but specificity 0.23. Consensus is therefore not a neutral target. The proposal accordingly anchors the cutoff to the diagnostic label and averages over raters, rather than treating the consensus contour as truth.

Core Claim

Let $b_r(x)$ be the biomarker value (here vCDR) computed from rater $r$'s tracing of image $x$, and let $y$ be the diagnosis. For a cutoff $t$, let $\mathrm{Sens}_r(t)$ and $\mathrm{Spec}_r(t)$ be the sensitivity and specificity of the rule $b_r(x) > t$. The usual practice picks $t$ to maximize Youden's index on consensus contours. We instead define the rater-marginal cutoff

$$t^\ast = \arg\max_t \; \mathbb{E}_{r}\big[\mathrm{Sens}_r(t) + \mathrm{Spec}_r(t) - 1\big],$$

which takes the expectation over the empirical panel of raters. We also report a robust variant that maximizes the worst rater's Youden index instead of the average. At deployment no raters are present, so we predict the crossing probability $p(x) = P_r\big(b_r(x) > t^\ast\big)$. The rule calls a case positive when $p(x) > 1-\alpha$, negative when $p(x) < \alpha$, and indeterminate otherwise.

The claim has three parts, each testable. First, cutoffs derived from different individual raters' contours differ by an amount that matters relative to automated vCDR error. Second, a cutoff derived on one rater's contours loses measurable sensitivity or specificity when applied to another rater's contours, and $t^\ast$ shrinks the worst-case loss. Third, $p(x)$ can be predicted from the image alone and is calibrated against the fraction of held-out raters whose value actually falls above $t^\ast$.

Research Program

The first step is purely descriptive and needs no new model. On REFUGE/REFUGE-2, we compute each specialist's vCDR for every image from their own disc and cup masks. We then derive a Youden cutoff separately for each specialist on the training split. We report the spread of these cutoffs, the rate at which a patient's call flips across specialists at a fixed cutoff, and the full rater-by-rater transfer matrix: sensitivity and specificity when the cutoff comes from rater $k$ and the test values come from rater $j$. This step depends on per-rater masks being available. REFUGE-2's multi-rater use in DiFF suggests they are; if only some splits have them, we restrict the analysis to those.

The REFUGE images come from two cameras. We will therefore fit a simple variance-components model to separate how much of the cutoff drift comes from raters and how much from devices. This links the rater question to the device shift that REFUGE observed.

The second step is the deployment-time predictor of $p(x)$. Several estimators will be compared on equal terms. The simplest is a heteroscedastic regressor: a network that outputs a mean and spread of rater vCDR, from which $p(x)$ follows under a Gaussian assumption. The second is a deep ensemble or test-time augmentation spread. The third is a Pionono-style latent-rater segmenter, whose sampled raters' masks are converted to vCDR values. We have no prior commitment to the generative model; if the regressor matches it, the regressor is the recommended method.

The third step tests generality on a second biomarker. We use nodule diameter and volume from the four LIDC-IDRI radiologists' contours against size-based management cutoffs. Here the endpoint is decision instability under the size rule itself, not malignancy, because Zhang et al. show that the malignancy labels are biased.

Baselines

The naive reference point is current practice. A cutoff is chosen by Youden's index on consensus-mask vCDR. At test time it is applied to vCDR computed from a single strong segmentation model's output, with no uncertainty and no indeterminate zone. Every metric below is reported for this reference.

The strongest existing competitor for the measurement itself is a well-tuned deterministic optic disc and cup segmenter in the style of the REFUGE leaders. The best reported vCDR error on REFUGE is MAE 0.0414 (team Masker, REFUGE), with optic cup Dice 0.8837. We cannot rerun Masker itself. We will therefore train our own U-Net under the same splits and report its MAE next to 0.0414, so readers can judge whether our baseline is competitive. This baseline is the main threat: if its error is far below the spread between raters, a single good model with a fixed cutoff might be all that is needed.

For uncertainty, the strongest competitor is a deep ensemble. The QUBIQ challenge reports that simple ensembles of deterministic models beat more complex probabilistic architectures. For calibration of crossing probabilities, no reported result found on Lacuna for any method.

All predictors share the same backbone, training images, augmentation, and hyperparameter search budget. The ensemble's member count is fixed so that its total training compute matches the latent-rater model's.

For diagnosis, we also report the end-to-end classifier references from REFUGE: AUC 0.9885 for VRT and 0.9471 for manual vCDR. We include DiFF's reported gain of up to 2.73% AUC on REFUGE-2 where comparable. We do not reimplement DiFF; its numbers serve as context for the AUC cost of keeping an explicit biomarker.

The key ablation crosses each uncertainty estimator with two cutoffs, the consensus-derived cutoff and $t^\ast$, each with and without the indeterminate zone. This separates the effect of the marginal cutoff from the effect of uncertainty estimation.

Beating the baselines means two things. The worst-rater drop in Youden's index, under held-out-rater transfer, should be smaller with $t^\ast$ than with the consensus cutoff, with the difference's confidence interval excluding zero under patient-level bootstrap. In addition, an indeterminate zone covering a fixed fraction of cases (for example 10%) should capture more of the cross-rater decision flips than an equally sized band of ±δ around the cutoff on the deterministic model's point estimate.

Why not just…?

Why not just use a better segmenter and the consensus cutoff? Mean absolute error is averaged over all patients, but decisions flip only for patients near the cutoff. A model with low average error can still give unstable calls for exactly those patients. Even so, this is a real threat. If the step-one flip rate and cutoff spread turn out small relative to 0.0414, the problem is minor for vCDR. In that case we would report that negative result and move the main claim to LIDC-IDRI nodule size, where four-radiologist contours on small nodules are more likely to straddle millimetre cutoffs.

Why not just take the spread of a deep ensemble or test-time augmentation? The QUBIQ findings make ensembles the uncertainty method to beat. They are included directly as one estimator of $p(x)$. Their open question is whether ensemble spread tracks disagreement between raters, which is a different quantity from model uncertainty. Calibration of $p(x)$ against held-out raters will settle this, and if ensembles are calibrated, they become the recommended estimator.

Why not just regress the mean and spread of rater vCDR directly? This may well work as well as a latent-rater generator, and it is far cheaper. For that reason the proposal does not depend on the generative model: the regressor is a first-class candidate. The contribution is the rater-marginal cutoff, the crossing-probability estimand, and the held-out-rater evaluation protocol, all of which hold whichever estimator wins.

Why not just drop the biomarker and use an end-to-end classifier, given 0.9885 against 0.9471 AUC? Clinicians and guidelines act on vCDR, a quantity they can check, and the DiFF authors themselves list the missing link to biomarkers as a limitation. A classifier's own score threshold is also derived on one cohort, so it inherits the drift problem rather than escaping it. We will report the AUC gap honestly so readers can weigh interpretability against accuracy.

Proposed Experiments and Decisive Checks

On REFUGE/REFUGE-2, the first decisive check is descriptive. We report the range of per-rater Youden cutoffs and the cross-rater flip rate. We set the bar for "material" in advance: the cutoff range exceeds twice the baseline segmenter's vCDR MAE. If it falls below that bar, the hypothesis fails for vCDR.

The second check uses leave-one-rater-out evaluation. The cutoff is derived without rater $j$ and tested on rater $j$'s contours, and we compare the worst-case Youden loss under $t^\ast$ against the consensus cutoff.

The third check measures calibration of $p(x)$ against the empirical fraction of held-out raters above the cutoff, using the Brier score and a reliability curve, across the three estimators.

The fourth check splits all results by camera, to test whether the rater-marginal cutoff remains useful once the device shift REFUGE documented is present.

One risk deserves stating directly. If REFUGE's glaucoma labels partly reflect how the cup looks to the same specialists, rater-marginal and consensus cutoffs could look artificially similar. We will check this by comparing results on the subset of cases with clinically confirmed labels, if that subset is identifiable.

On LIDC-IDRI, we repeat the flip-rate and leave-one-radiologist-out analyses for diameter and volume cutoffs. Success on both datasets would support a simple reporting standard for any image-derived biomarker: publish the cutoff's spread across raters alongside the cutoff itself.