Likelihood-based Nonparametric Receiver Operating Characteristic Curve Analysis in the Presence of Imperfect Reference Standard

Yifan Sun, Peijun Sang, Qinglong Tian, Pengfei Li

TL;DR

Researchers have developed a new likelihood-based, nonparametric method to evaluate diagnostic tests when the "truth" (the reference standard) is prone to errors. By using a density ratio model and an efficient EM algorithm, this approach provides more accurate and smoother ROC curves than previous methods, even when labels are misclassified.

What Problem Does This Paper Address?

In medical diagnostics, the Receiver Operating Characteristic (ROC) curve is the bread and butter of performance evaluation. It plots sensitivity against specificity to show how well a biomarker (like a blood protein level) distinguishes between diseased and healthy individuals.

Standard ROC analysis assumes we have a "gold standard"—a perfectly accurate way to know who is actually sick. However, in the real world, gold standards are often expensive, invasive, or non-existent. Instead, researchers use "imperfect reference standards" (like a cheaper, less accurate test).

If you treat an imperfect standard as if it were perfect, your ROC analysis becomes biased, usually underestimating how good your new biomarker actually is. While some methods exist to correct for this, they often struggle with "noisy" results or fail to account for the fact that biomarkers usually follow smooth, continuous distributions.

Methods

The authors propose a likelihood-based nonparametric approach. Here’s the breakdown of their "secret sauce":

  1. Density Ratio Modeling: Instead of trying to estimate the full distribution of the biomarker for healthy and diseased groups separately, they focus on the log density ratio—the relative "weight" of being diseased versus healthy at any given biomarker value.
  2. Sieve Estimation (B-splines): To keep the model flexible (nonparametric) but manageable, they approximate this ratio using B-spline basis functions. This allows the model to adapt to the data's shape without assuming it follows a specific curve like a Normal distribution.
  3. The EM Algorithm: Since the true disease status is unobserved (it's "missing data"), they use an Expectation-Maximization (EM) algorithm. It iteratively guesses the true status (E-step) and then updates the model parameters to maximize the likelihood of the observed data (M-step).
  4. Roughness Penalty: They include a penalty term to ensure the resulting ROC curve is smooth and doesn't "wiggle" too much due to random noise in small samples.

Datasets

The researchers validated their method using two primary sources:

  • Extensive Simulations: They generated synthetic data using univariate and bivariate Normal and Gamma distributions. They tested various sample sizes ($n=100$ to $500$) and different levels of reference standard accuracy.
  • Malaria Study: They applied the method to real-world data from a study in Tanzania. The goal was to evaluate how well parasite levels in the blood could diagnose malaria, using "season" (wet vs. dry) as an imperfect reference standard for malaria prevalence.

Key Insights and Ideas

  • Continuity is a Feature, Not a Bug: Previous nonparametric methods (like the one by Sun et al., 2024) used an "inversion method" that didn't fully exploit the continuous nature of biomarkers. By using splines, this new method produces much smoother, more biologically plausible ROC curves.
  • Information Pooling: The likelihood-based approach effectively "borrows" information from both the nominal case group and the nominal control group simultaneously. This leads to significantly lower Mean Squared Error (MSE) in estimations.
  • Identifiability: The paper reinforces that you can recover the true ROC curve from imperfect data, provided you have some prior knowledge about the accuracy of your reference standard (specifically, the probability that a "nominal" case is a "true" case).

Key Results

  • Superior Accuracy: In simulations, the proposed EM-based method consistently showed smaller MSE for the ROC curve, the Area Under the Curve (AUC), and the Youden’s index compared to existing nonparametric techniques.
  • Reduced Bias: While "naive" methods (treating the imperfect standard as truth) were heavily biased, this new method successfully recovered the true parameters.
  • Real-World Utility: In the malaria study, the proposed method produced a smoother ROC curve and a more reliable Youden’s index than the previous state-of-the-art, which tended to overestimate the index.

Limitations and Future Work

  • Known Accuracy Parameters: The model assumes that the accuracy of the reference standard ($\pi_0$ and $\pi_1$) is known or can be estimated from auxiliary data. If these estimates are wrong, the model's results will be affected.
  • Computational Intensity: Using B-splines with cross-validation for tuning parameters and an iterative EM algorithm is more computationally demanding than simpler "plug-in" empirical estimators.
  • Future Directions: The authors suggest that extending this framework to handle covariate adjustments (like age or sex) or longitudinal data would be a logical next step for the field.

Summary updated

Abstract

In diagnostic studies, researchers frequently encounter imperfect reference standards with some misclassified labels. Treating these as gold standards can bias receiver operating characteristic (ROC) curve analysis. To address this issue, we propose a novel likelihood-based method under a nonparametric density ratio model. This approach enables the reliable estimation of the ROC curve, area under the curve (AUC), partial AUC, and Youden's index with favorable statistical properties. To implement the method, we develop an efficient expectation-maximization algorithm algorithm. Extensive simulations evaluate its finite-sample performance, showing smaller mean squared errors in estimating the ROC curve, partial AUC, and Youden's index compared to existing methods. We apply the proposed approach to a malaria study.
BibTeX
@inproceedings{sun2025likelihood,
  author = {Yifan Sun and Peijun Sang and Qinglong Tian and Pengfei Li},
  title = {{Likelihood-based Nonparametric Receiver Operating Characteristic Curve Analysis in the Presence of Imperfect Reference Standard}},
  booktitle = {Statistics},
  year = {2025},
  doi = {10.1002/sim.70327},
}

Loading related content...

Figure