Clinical risk models often take a number measured from an image, such as a nodule's volume, a cup-to-disc ratio, or a texture score, and feed it into a multivariable logistic regression. That is a model that estimates how much each factor raises the odds of disease while holding the other factors fixed. The number is treated as exact. It is not. It comes from an outline that one particular expert drew, and a second expert would have drawn it differently. This note proposes to measure and correct the damage that this disagreement does to the reported odds ratios. Statisticians call the problem measurement error. The key point is that this error may not be random noise. When the same radiologist both traces the lesion and rates how suspicious it looks, a nodule that looks suspicious may get a wider outline. The image feature and the label then share a hidden common cause, the rater's suspicion. This inflates the apparent association rather than diluting it, and averaging more raters cannot remove it.
Concretely, we treat the multiple expert outlines that public benchmarks already provide as repeated measurements of the same quantity. We split the image-derived feature into three parts: the true value, a rater-specific offset, and a part that moves with the outcome. We then build a correction for the logistic model that uses these repeated outlines. To handle the much more common case where a clinical cohort has only one outline per patient, we generate synthetic repeated outlines with rater-conditional segmentation models. The deliverable is threefold: an audit of how much published-style odds ratios move when one expert's outline is swapped for another's, a test of whether that movement is outcome-coupled, and a correction whose performance can be checked against a known ground truth.
Core Claim
Image features are routinely fed into logistic nomograms (point-score charts built from a logistic model). We claim that, in these features, inter-observer variability has two separable components with opposite effects on odds ratios. The first is ordinary random scatter, which biases odds ratios toward 1. The second is a suspicion-coupled component, which biases them away from 1 whenever the outline and the label come from the same reader. In a simple measurement model, rater $r$'s feature for case $i$ is
$$W_{ir} = X_i + b_r + \gamma\, S_{ir} + u_{ir},$$
where $X_i$ is the true feature, $b_r$ is rater $r$'s systematic offset, $S_{ir}$ is that rater's own suspicion or diagnostic score for the case, $\gamma$ measures how much suspicion shifts the outline, and $u_{ir}$ is residual scatter. Classical error theory assumes $\gamma = 0$. Under that assumption, a logistic coefficient fit on the mean of $R$ raters is shrunk by roughly
$$\lambda = \frac{\operatorname{var}(X)}{\operatorname{var}(X) + \operatorname{var}(u)/R},$$
because the regression slope scales with the fraction of feature variance that is signal. When $\gamma \neq 0$ and the label is built from the same $S_{ir}$, no value of $R$ removes the bias. The new contributions are a direct within-case estimate of $\gamma$ on real multi-rater data, and a correction that handles both terms.
Relation to Prior Work
Several studies motivate the gap. Chen et al.'s thyroid CT nomogram combined five lasso-selected radiomic features from single manual outlines with clinical factors. They acknowledged that boundary shifts change textures but never measured how this affected their coefficients. Zhang et al.'s pituitary adenoma study deliberately expanded hand-drawn masks to capture the tumor edge. They also named manual outlining as the main source of variability without propagating it. The LIDC-IDRI surrounding-structure study fit logistic classifiers on nodule-to-vessel distances using malignancy labels taken from the same radiologists' ratings. This is exactly the shared-reader design in which we expect suspicion coupling. Zhang, Gu et al. showed that LIDC-IDRI ratings carry a "safety-first" malignancy bias: models trained on consensus labels reached sensitivity 0.97 but specificity 0.23. They studied only the scalar score, never the outlines. Our hypothesis is that the same defensive tendency also appears in the geometry.
Other work supplies methods we reuse. Rater-conditional segmentation models learn how each expert would outline a structure. Pionono gives each rater a Gaussian in a latent space, whose mean is the rater's systematic style and whose spread is the rater's inconsistency. Liu et al.'s variational model, OMBA-UNet and MoSE sample diverse plausible masks. All four stop at pixel-level scores such as Dice, generalized energy distance, or kappa. None of them asks what their samples do to a downstream odds ratio. We use them as generators of synthetic repeated outlines, and our evaluation is the regression output, not mask overlap. Huang et al.'s weighted lasso gives non-asymptotic guarantees for sparse logistic regression when covariates carry bounded measurement error. Its error is a generic, non-differential fluctuation, and its target is variable selection. It is the right tool for the high-dimensional radiomics arm, but it does not address outcome-coupled error or estimate odds ratios.
One paper constrains the design. DiFF argues that the "right" outline is whichever mix of raters best predicts diagnosis. That idea is attractive for prediction. It is dangerous for odds-ratio estimation, because optimizing outlines to predict the label can deliberately build in the very coupling we warn about. We address this directly by using DiFF-calibrated masks as one feature source and testing whether they inflate odds ratios. Finally, QUBIQ reported that simple ensembles of deterministic U-Nets often beat probabilistic architectures at matching rater disagreement. This warns us that the generative step may not need elaborate models, so we include a cheap ensemble generator.
Research Program
The work has three stages.
Audit. On multi-rater datasets with binary outcomes, we compute each image feature from every individual rater's outline. We then refit the full multivariable logistic model once per rater and report the spread of adjusted odds ratios, confidence-interval coverage, and sign flips. We use variance-component models to split feature variance into between-case, rater-offset, and residual parts. Where rater identities are not consistent across scans, as in LIDC-IDRI, rater offsets become case-level random effects.
Coupling test. In LIDC-IDRI each of up to four readers supplies both an outline and a malignancy score for the same nodule. We regress a reader's outline-derived volume, diameter, or vessel distance on that reader's own score, with a fixed effect for each nodule. The fixed effect absorbs everything true about the nodule, so any remaining slope estimates $\gamma$. REFUGE-2 serves as a negative control: its glaucoma labels are a clinical diagnosis rather than a per-grader rating, so the coupling should be absent there.
Correction. We combine three ingredients. The first is leave-rater-out pairing: features come from one rater's outline and labels from the other raters, which cuts the shared-reader path. The second is replicate-based regression calibration with rater random effects, which replaces each feature with its predicted true value given the repeated outlines. The third is a resampling-based simulation-extrapolation step, which deliberately adds more rater-like noise and extrapolates back to zero noise. For single-outline cohorts, a rater-conditional generator trained on a multi-rater source supplies synthetic repeated outlines. Candidate generators are Pionono-style latents and, as the cheap option, a U-Net ensemble. The regression-calibration step then uses these synthetic replicates in place of real ones. For high-dimensional radiomic signatures, the corrected features feed into Huang et al.'s Type II weighted lasso. Its per-feature weights are set from the measured rater variance rather than from the raw feature scale.
Baselines
The do-nothing reference fits the multivariable logistic model on features from a single rater's outline, picked at random. This is how Chen et al. and most nomogram studies proceed. For context, Chen et al. report validation AUC 0.812 for their radiomic nomogram against 0.756 for the clinical-only model. No reported result was found on Lacuna for how much their odds ratios change across raters, and quantifying exactly that is the point.
The second baseline is the consensus approach: features from a majority-vote or STAPLE mask, or the per-case mean of the rater features. With four raters it is the strongest simple competitor for the random-scatter component.
The third baseline is per-rater refit with pooled estimates. We fit once per rater and combine the estimates as if they were multiply imputed datasets. This honestly widens the intervals but leaves the point estimate attenuated.
The fourth is reliability filtering. Features whose intraclass correlation across raters falls below a threshold are dropped before lasso selection. This is the default radiomics practice.
For the high-dimensional arm, the strongest existing method is Huang et al.'s Type II weighted lasso. Its reported result is zero refitting prediction error with 8 to 29 selected genes on the Golub leukemia data, but no reported result was found on Lacuna for it on imaging features.
As a prediction-only comparator we include DiFF. It reports diagnostic AUC gains of up to 2.73% on REFUGE-2, 2.51% on TNMIX and 2.04% on ISIC over consensus-trained models. We use its calibrated masks as a feature source.
All methods share the same patients, the same splits, and the same covariate set. Every method that needs a tuning parameter gets the same cross-validation grid.
Three ablations isolate the key ingredients. The first compares leave-rater-out pairing with and without calibration. The second compares calibration using real versus synthetic replicates. The third compares a Pionono-style generator with the U-Net ensemble generator.
The criterion for beating the baselines is set on semi-synthetic ground truth. The corrected estimator must recover the true log-odds ratio with lower bias and near-nominal 95% interval coverage, while the consensus baseline shows measurable bias. On real data, a further criterion applies wherever an external label exists that is independent of all raters. The corrected odds ratio must agree with the estimate obtained using that independent label better than the naive estimate does.
Why not just…?
Why not just average the raters' features, or use a STAPLE consensus? For purely random scatter with four raters, averaging cuts the error variance by a factor of four. It may be nearly as good as our correction, and that is a real threat. Averaging cannot remove a rater offset shared across cases, and it does nothing to the suspicion-coupled term, because every rater's own score pushes in the same direction. It is also unavailable in the typical single-outline cohort. The semi-synthetic experiment settles this directly: we vary $\gamma$ from zero upward and locate the point where averaging stops being adequate. If it stays adequate at the $\gamma$ we measure on LIDC-IDRI, we will report that. The contribution then narrows to the audit and to the single-outline extension.
Why not just drop unreliable features by intraclass correlation? Filtering discards information instead of correcting it. More importantly, a feature can be highly reproducible across raters and still be suspicion-coupled, because coupling is a correlated shift rather than scatter. We test this by checking whether features that pass the filter still show a nonzero within-nodule $\gamma$.
Why not just apply the weighted lasso of Huang et al.? It is the right tool for choosing features under bounded, outcome-independent noise, and we adopt it for that role. Its guarantees concern selection and prediction error, not unbiased odds ratios. They also assume the error is unrelated to the outcome, which is the assumption we are testing. Running it with and without our corrected inputs shows whether it is sufficient by itself.
Why not just replace manual outlines with one deterministic U-Net? A single network gives repeatable outlines, but repeatable is not the same as unbiased. A network trained on rater outlines inherits their average offset and any coupling. Moreover, a single fixed outline provides no information about the size of the error, so attenuation cannot be estimated. The same QUBIQ evidence that favors U-Net ensembles is why we use an ensemble as one of our replicate generators rather than a single network.
Proposed Experiments and Decisive Checks
Semi-synthetic recovery on REFUGE-2. We take the consensus vertical cup-to-disc ratio as the "true" $X$ and add a simulated age-like confounder. We generate outcomes from a logistic model with known coefficients, then fit every method using individual graders' outlines. Because the truth is known, this measures bias, coverage, and the attenuation factor $\lambda$ exactly. We repeat it while injecting synthetic suspicion coupling at controlled $\gamma$.
Real coupling test on LIDC-IDRI. We estimate $\gamma$ with a fixed effect per nodule for volume, diameter, and nodule-to-vessel distance. We then rerun the surrounding-structure logistic analysis with same-reader versus leave-rater-out pairing. As a partly independent check, we also use the Zhang, Gu et al. relabeled malignancy labels, which are anchored in pathology. The decisive outcome is a nonzero within-nodule $\gamma$ together with odds ratios that shrink when the shared-reader path is cut. A null $\gamma$ would falsify the coupling half of the claim.
Synthetic-replicate transfer. We train generators on LIDC-IDRI, then hide all but one outline per case. We check whether calibration using synthetic replicates recovers the estimate obtained with all real replicates. The same check is run on TNMIX and ISIC, which DiFF used with multiple raters. Outcomes there are as provided in those datasets. No reported result was found on Lacuna for odds-ratio stability on any of these benchmarks, so all comparisons are internal.
DiFF features versus consensus features. Finally, we test whether DiFF-calibrated mask features yield larger odds ratios than consensus features in the shared-reader setting but not in the leave-rater-out setting. If so, that would show that diagnosis-optimized outlines carry the coupling into the regression.
References
- Computed Tomography Radiomic Nomogram for Preoperative Prediction of Extrathyroidal Extension in Papillary Thyroid Carcinoma
- Prediction of high infiltration levels in pituitary adenoma using MRI-based radiomics and machine learning
- Relationship between pulmonary nodule malignancy and surrounding pleurae, airways and vessels: a quantitative study using the public LIDC-IDRI dataset
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- Weighted Lasso Estimates for Sparse Logistic Regression: Non-asymptotic Properties with Measurement Error
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge