Likelihood‐Based Non‐Parametric Receiver Operating Characteristic Curve Analysis in the Presence of Imperfect Reference Standard

Yifan Sun, Peijun Sang, Qinglong Tian, Pengfei Li

TL;DR

This paper introduces a novel likelihood-based, non-parametric method for analyzing Receiver Operating Characteristic (ROC) curves when the "gold standard" for disease status is imperfect. By leveraging a density ratio model and an Expectation-Maximization (EM) algorithm, their approach offers more accurate and smoother ROC curve estimations, outperforming existing methods in simulations.

What Problem Does This Paper Address?

In medical diagnostics, we often use biomarkers to distinguish between healthy and diseased individuals. The Receiver Operating Characteristic (ROC) curve is a standard tool for evaluating how well a biomarker performs this task. Traditionally, ROC analysis assumes we have a perfect "gold standard" to definitively label individuals as diseased or healthy. However, in many real-world scenarios, this gold standard is unavailable or imperfect, meaning some individuals are misclassified. Using an imperfect reference standard as if it were perfect can lead to biased and less reliable ROC curve analyses. This paper tackles the challenge of performing accurate ROC analysis when only an imperfect reference standard is available, aiming to provide a more robust statistical framework.

Methods

The core of this paper's contribution lies in a new likelihood-based, non-parametric approach. Instead of directly modeling the biomarker distributions for diseased and healthy groups (which is difficult with an imperfect reference), they model the ratio of their probability density functions. This "density ratio" is then approximated using B-spline basis functions, a flexible non-parametric technique that can capture complex shapes.

To estimate the parameters of this model and subsequently the ROC curve, AUC, partial AUC, and Youden's index, they develop an efficient Expectation-Maximization (EM) algorithm. The EM algorithm is particularly well-suited for problems with missing data (in this case, the true disease status is "missing" due to the imperfect reference). A key advantage is that the E-step has a closed-form solution, and the M-step can be efficiently solved using existing R packages, making the implementation practical. They also incorporate a roughness penalty into the likelihood function to ensure smoother estimates, addressing a limitation of previous methods.

Datasets

The paper evaluates its proposed method using simulated datasets and a real-world malaria dataset.

  • Simulated Data: The simulations involve both univariate and bivariate biomarkers generated from Gaussian and Gamma distributions under various misclassification scenarios (different levels of sensitivity and specificity for the imperfect reference standard). These simulations are designed to mimic real-world diagnostic scenarios with imperfect labels.
  • Malaria Dataset: The authors apply their method to a dataset from a cross-sectional survey on malaria in Tanzania. This dataset is used to assess the performance of parasite levels in blood as a biomarker for diagnosing malaria, where the reference standard is based on the season (wet vs. dry) rather than a definitive diagnosis.

Key Insights and Ideas

  • Density Ratio Modeling is Powerful: Instead of directly estimating the distributions of biomarker values for diseased and healthy individuals (which is complicated by imperfect labels), modeling the ratio of these densities proves to be a more tractable and effective approach. This is a key conceptual shift.
  • EM Algorithm for Robustness: The use of an EM algorithm is crucial for handling the "missing" true disease labels caused by the imperfect reference standard. It allows for robust estimation even when the reference standard is not perfectly accurate.
  • Smoothness Matters: Incorporating a roughness penalty in the density ratio model leads to smoother ROC curves compared to existing non-parametric methods. This is important because smoother curves often indicate more reliable estimates and can better reflect the underlying continuous nature of biomarkers.
  • Likelihood-Based Approach Offers Efficiency: The paper argues that a likelihood-based framework, by definition, leverages all available information in the data more efficiently than inversion-based methods, potentially leading to better statistical properties.

Key Results

  • Improved Accuracy: In simulation studies, the proposed likelihood-based method consistently demonstrated smaller mean squared errors (MSEs) for estimating the ROC curve, partial AUC, and Youden's index compared to the fully non-parametric method proposed by Sun et al. (2024).
  • Reduced Bias: The bias in estimating Youden's index was substantially smaller with the proposed method compared to the fully non-parametric approach.
  • Smoother ROC Curves: The real-data analysis showed that the proposed method produces smoother ROC curves than the existing non-parametric method, aligning with the expectation of incorporating a roughness penalty.
  • Comparable AUC Performance: For estimating the Area Under the Curve (AUC), the proposed method showed similar performance to the existing non-parametric method.
  • Outperforms Naive Method: Both the proposed and the fully non-parametric methods significantly outperformed a "naive" method that incorrectly treats the imperfect reference standard as a gold standard.

Limitations and Future Work

  • Tuning Parameter Selection: The performance of the proposed method relies on selecting appropriate tuning parameters, such as the number of B-spline basis functions (K) and the smoothing parameter (ν). While cross-validation is used, the sensitivity to these choices could be a consideration.
  • Assumptions: The method relies on assumptions such as the known accuracy of the reference standard (π₀ and π₁) and conditional independence between the reference standard and the biomarker given the true disease status. The paper notes that these parameters can be estimated from auxiliary data, but their accurate estimation is crucial.
  • Future Work: The paper suggests that the developed framework could be extended to handle multivariate biomarkers more comprehensively and to explore more complex scenarios of imperfect reference standards. The authors also highlight the potential for further theoretical investigations into the asymptotic properties of all estimated indices.

Summary updated

Abstract

In diagnostic studies, researchers frequently encounter imperfect reference standards with some misclassified labels. Treating these as gold standards can bias receiver operating characteristic (ROC) curve analysis. To address this issue, we propose a novel likelihood-based method under a non-parametric density ratio model. This approach enables the reliable estimation of the ROC curve, area under the curve (AUC), partial AUC, and Youden's index with favorable statistical properties. To implement the method, we develop an efficient expectation-maximization (EM) algorithm. Extensive simulations evaluate its finite-sample performance, showing smaller mean squared errors in estimating the ROC curve, partial AUC, and Youden's index compared to existing methods. We apply the proposed approach to a malaria study.
BibTeX
@inproceedings{sun2025likelihoodbased,
  author = {Yifan Sun and Peijun Sang and Qinglong Tian and Pengfei Li},
  title = {{Likelihood‐Based Non‐Parametric Receiver Operating Characteristic Curve Analysis in the Presence of Imperfect Reference Standard}},
  booktitle = {Statistics},
  year = {2025},
  doi = {10.1002/sim.70327},
}
Figure