Measuring Each Expert's Diagnostic Threshold with Synthetic Image Ladders

Experts who look at the same scan often disagree, and much of that disagreement comes from where each one draws the line between "benign" and "malignant." Current methods try to infer each expert's line from the natural cases they happened to label. Most of those cases are far from anyone's line, so the estimates are noisy, and the cohorts are too small to correct the noise. We propose a different route. Generative models can now produce scans that radiologists cannot tell from real ones. We use such a model to build ladders: sequences of synthetic images of one nodule that change step by step along an axis calibrated to biopsy outcomes. Each expert reads a short ladder, and the point where they switch from "benign" to "malignant" is a direct measurement of their personal threshold.

Those measured thresholds then become a tool against overfitting in small cohorts. A rater's score on a real case can be turned into a statement about where the case lies on the biopsy-anchored axis. This lets a classifier train on labels with rater bias removed, so it does not memorize rater idiosyncrasies. Put in two sentences: realistic synthesis turns a diagnostic image into a controlled stimulus, so a few dozen targeted reads pin down each expert's threshold better than hundreds of ordinary reads. Knowing those thresholds lets a small, multi-rater cohort be relabeled on a common, pathology-grounded scale.

Core Claim

The claim is that expert-level synthetic realism is most valuable as a measurement instrument rather than as extra training data. The ladder method is borrowed from psychophysics, the study of how perception responds to controlled changes in a stimulus. Let $t$ be a stimulus's position on a biopsy-calibrated malignancy axis. We model rater $r$'s probability of calling it malignant as

$$ p_r(\text{malignant} \mid t) = \sigma\!\left(\frac{t - \theta_r}{s_r}\right), $$

where $\sigma$ is the logistic function, $\theta_r$ is the rater's threshold (their systematic bias), and $s_r$ is their slope width (their inconsistency from read to read). For a logistic response, the information one read carries about $\theta_r$ is proportional to $p(1-p)$. That quantity is largest at $t = \theta_r$ and falls off quickly away from it. Natural cohorts place most cases far from the boundary, so they carry little information per read. An adaptive ladder can place nearly every stimulus near the boundary. Our quantitative expectation is that, at equal reading time, ladder reads give a several-fold smaller error on $\theta_r$ than natural-case reads. The exact factor depends on how much of a natural cohort falls near each rater's threshold, and we measure that factor empirically.

This parameterization echoes Pionono, which models each segmentation rater as a Gaussian in latent space with mean for bias and variance for inconsistency. The difference lies in what the parameters refer to. Pionono's latent space is unanchored and learned from existing labels. Our $\theta_r$ is a location on an axis tied to pathology, and it is estimated from new, designed observations.

Why This Gap, and Not the Obvious One

The obvious idea at this intersection is to generate images conditioned on rater identity, or to synthesize ambiguous cases as augmentation. Both inherit a circularity: the labels for the synthetic images come from models fit to the same small labeled set. A skeptic would reasonably say that pseudo-labeling unlabeled real images, or mixing features as in MixUp-MIL (Koller), captures most of that benefit. We therefore steered away from it. The ladder design adds information that did not previously exist, namely human responses to controlled stimuli.

Four lines of prior work motivate the design. Realism results show the stimuli can be credible. Radiologists were at 50.4% on synthetic chest X-rays and undecided in 41.8% of trials (Perceptual Evaluation of GANs and Diffusion Models for Generating X-rays). They mistook synthetic CT tumors for real up to 45% of the time (Text-Driven Tumor Synthesis).

The source of the method is the retinal counterfactual work. It steers an unconditional diffusion model toward a diagnosis using gradients from an adversarially robust classifier, mixed with standard gradients through a cone projection. Experts were near chance on its fundus counterfactuals. We reuse that steering mechanism, but our guiding classifier is trained on biopsy truth rather than on clinical grades.

The problem itself comes from Re-thinking and Re-labeling LIDC-IDRI. That paper found that over half of nodules radiologists scored "likely malignant" were benign on pathology. It corrected this by propagating labels from 180 biopsy-proven SCH-LND nodules through metric learning, moving sensitivity/specificity from 0.97/0.23 to 0.67/0.66. That correction is applied to the whole panel at once. It never asks how much each radiologist over-calls, and it discards the raters' information instead of reusing it.

Finally, the rater-modeling methods infer style from existing labels on natural cases. These are the Latent Doctor Model, with a set encoder over a support set of a doctor's past reads, and the virtual expert panel, with one classifier per pathologist. Both treat the panel distribution as the target rather than anchoring it to pathology. The virtual panel work also excludes "indefinite" cases, which are exactly the ones a ladder targets.

One finding constrains the design. Min et al. report that synthetic images can look more realistic than real ones because they omit artifacts. Raters might therefore respond to synthetic stimuli differently than to real ones. We treat that possibility as a primary validity check rather than an afterthought.

Research Program

The study uses thoracic CT nodules, where both a multi-rater cohort (LIDC-IDRI, 1–5 malignancy scores) and a biopsy-anchored set (SCH-LND, 180 nodules, 90 benign and 90 malignant) exist.

The first component is the generator. We train a 3D latent diffusion model on LIDC-IDRI nodule patches and surrounding unlabeled CT, cropped to a region of interest as in Diffusing the Blind Spot. We apply that paper's memorization filter, discarding samples with cosine similarity above 0.95 to training patches.

The second component is the axis. We train an adversarially robust malignancy classifier on SCH-LND only. We define $t$ as its calibrated logit, using cross-fitting so that the nodules later used for evaluation never shape the axis. From a seed nodule, guided sampling with cone projection produces a ladder of perhaps 15 images with fixed anatomy and monotone $t$. A held-out non-robust classifier and a radiologist review screen out ladders where the axis changes nodule size alone. Size is a confound that would otherwise be too easy a cue.

The third component is the reading study. We recruit a panel of about eight thoracic radiologists, since the original LIDC-IDRI readers are anonymous. Each reads adaptive ladders chosen by a Bayesian staircase that places the next stimulus near the current estimate of $\theta_r$. Ladders are interleaved with real "catch" nodules whose $t$ values match, so threshold consistency on real and synthetic stimuli can be compared within each reader. The same readers also score a natural set of real nodules, both to fit the comparison arm and to supply labels for relabeling.

The fourth component is relabeling and training. Each real-case score from rater $r$ is inverted through that rater's fitted response curve, giving a posterior over $t$ for the case. These posteriors are pooled across raters into a bias-corrected soft label. A malignancy classifier is then trained on the relabeled cohort, using frozen diffusion features with a small probe, following FEDD to limit overfitting.

Before any human reading, we run a simulation. Virtual raters with known thresholds read both ladders and natural cases. If ladder estimates are not clearly more efficient in silico, the human study does not proceed.

Baselines

The naive reference point is majority-vote consensus on LIDC-IDRI scores, binarized at score 4 or above. Its reported profile is sensitivity 0.97 and specificity 0.23 (Re-thinking and Re-labeling LIDC-IDRI). We re-run it with our classifier so the comparison is like for like.

The strongest existing method is the metric-learning relabeling from that same paper, which reports sensitivity 0.67 and specificity 0.66. We re-implement it on the same LIDC-IDRI cohort with the same SCH-LND cross-fitting folds, the same frozen-feature classifier, and the same hyperparameter search budget. Beating it means matching its sensitivity with significantly higher specificity, or matching its specificity with higher sensitivity, on held-out SCH-LND folds, together with a higher AUC whose bootstrap interval excludes it.

The strongest rater-modeling competitor is the Latent Doctor Model. It infers each new reader's style from their natural-case reads and is trained on identical data. We find no reported result on Lacuna for it on LIDC-IDRI; its strongest reported figure elsewhere is a Spearman correlation of up to 0.728 with 20-pathologist entropy on ISUP.

The control that isolates our key ingredient is an equal-reading-time arm. Here each radiologist spends the ladder time on additional, actively selected real nodules, and the same response-curve fit and relabeling are applied. Two further ablations test what the ladder depends on. One replaces the biopsy-calibrated axis with an axis guided by consensus grades. The other replaces adaptive stimulus placement with uniformly spaced ladders.

Why not just…?

Why not just train on majority-vote consensus? Consensus inherits the panel's shared bias. On LIDC-IDRI it gives 0.23 specificity, and on Barrett's grading consensus training was both less accurate and far worse calibrated than a full panel (F1 72.9% versus 78.2%, calibration error 0.183 versus 0.050; More is more). It remains in the study only as the floor.

Why not just relabel with biopsy anchors, as the LIDC-IDRI re-labeling paper already does? This is the most serious threat. If panel-level metric learning already captures most of the correctable bias, per-rater thresholds may add little to downstream accuracy. The head-to-head comparison above settles this. If our method fails to beat 0.67/0.66, we will report that per-rater measurement mainly improves calibration and interpretability rather than accuracy. That would be a narrower but still useful result.

Why not just estimate rater bias from existing labels, with the Latent Doctor Model, Pionono, or per-rater virtual panels? These methods identify bias only relative to the panel and only from cases near each rater's boundary, which are scarce in small cohorts. They are run as baselines. Their threshold estimates are also compared against ladder estimates on held-out real catch trials, which shows directly which method better predicts a rater's next real call.

Why not just spend the reading budget on more real cases? If actively selected real nodules near the boundary are plentiful, synthetic ladders are unnecessary. The equal-reading-time arm tests this directly. We expect ladders to win mainly because real nodules cannot vary one malignancy cue while holding anatomy fixed. If they do not win, that is the honest conclusion.

Decisive Checks

Three results decide the proposal. The first is validity: within each reader, thresholds fit on synthetic ladders must agree with thresholds fit on $t$-matched real catch nodules, with the difference statistically indistinguishable from zero. If raters judge idealized synthetic nodules differently, as Min et al. warns they might, the instrument fails. The second is efficiency: at equal reading time, ladder-derived $\theta_r$ must predict held-out real calls better than estimates from natural reads. The third is utility: the relabeled classifier must beat metric-learning relabeling on held-out biopsy-proven SCH-LND nodules. Given only 180 such nodules, all three are reported with bootstrap confidence intervals.

If the method succeeds, a natural replication is Barrett's dysplasia on DEPP, where the excluded "indefinite" cases would serve as ladder seeds.