Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface

Jialiang Li, Xiao‐Hua ZhouJournal of Statistical Planning and Inference 2009 · DOI

TL;DR

This paper introduces new statistical methods for evaluating diagnostic tests that classify subjects into three ordered categories, moving beyond traditional binary classification. They develop both nonparametric and semiparametric approaches to estimate the "ROC surface," a visualization of diagnostic accuracy, and introduce the concept of Volume Under the ROC Surface (VUS) as a single summary measure.

What Problem Does This Paper Address?

Many real-world diagnostic scenarios involve more than a simple "diseased" or "healthy" classification. For instance, disease severity can be graded (e.g., mild, moderate, severe dementia). Traditional Receiver Operating Characteristic (ROC) curve analysis, designed for binary outcomes, struggles with these multi-category situations. Simply forcing multi-category data into a binary framework can lead to biased accuracy estimates. This paper addresses the need for robust statistical tools to accurately assess diagnostic tests with ordered, multi-category outcomes, specifically focusing on the three-category case.

Methods

The authors propose two main estimation approaches:

  • Nonparametric Estimation: This method directly uses the observed data to estimate the ROC surface without making strong assumptions about the underlying distributions of test results for each class. They replace theoretical distribution functions with their empirical counterparts, essentially building the ROC surface from the data itself. This approach is theoretically grounded with proofs of consistency and asymptotic normality.
  • Semiparametric Estimation: This approach assumes a specific functional form for the ROC surface, motivated by the assumption that test results follow normal distributions. However, it allows for flexibility by not requiring exact knowledge of these distributions. They propose two methods for fitting this parametric model, which involve solving systems of non-linear equations.

A key concept introduced is the Volume Under the ROC Surface (VUS), which is a generalization of the area under the 2D ROC curve. VUS provides a single, interpretable value summarizing the overall diagnostic accuracy across all three classes.

Datasets

The paper uses a real-world dataset from a study on Alzheimer's disease (AD). This dataset includes 3728 subjects, classified into three disease stages (high, intermediate, and low likelihood of AD) based on post-mortem autopsy. Test results from the Mini-Mental State Examination (MMSE) were used as the diagnostic test. The authors also conducted extensive simulation studies using data generated from normal distributions to evaluate their methods.

Key Insights and Ideas

  • Beyond Binary: The paper highlights the limitations of traditional binary ROC analysis for multi-category diagnostic problems and provides a principled way to extend it.
  • ROC Surface as a Tool: The ROC surface is presented not just as a theoretical construct but as a practical graphical tool to visualize and understand diagnostic accuracy across different thresholds for each of the three classes.
  • VUS as a Summary: The Volume Under the ROC Surface (VUS) offers a valuable single metric for overall diagnostic performance, analogous to AUC in binary settings, and is shown to be equivalent to the probability of correctly ordering three randomly selected subjects from the three classes.
  • Nonparametric vs. Semiparametric Trade-offs: The nonparametric method is robust to distributional assumptions but can be more complex to interpret and visualize. The semiparametric method offers a smooth, interpretable surface but relies on the chosen functional form and distributional assumptions (though it can be adapted for other families).
  • Importance of Distributional Assumptions: Simulation results suggest that the validity of the semiparametric model is sensitive to the underlying distributional assumptions, with normality performing well, but other distributions (Weibull, gamma) leading to bias and inefficiency.

Key Results

  • Simulation Studies: Both nonparametric (Brownian bridge) and semiparametric methods performed well in simulations. The semiparametric method (Method 2) generally produced more efficient estimates and better coverage rates for confidence intervals, especially at smaller sample sizes.
  • Alzheimer's Data Analysis:
    • The nonparametric ROC surface was estimated and visualized, showing the trade-offs in classification accuracy for the three AD stages.
  • The estimated VUS for the AD dataset was 0.39, with a 95% confidence interval of [0.37, 0.41], suggesting about a 40% probability of correctly classifying three random subjects from the population.
  • Semiparametric models were fitted, yielding smooth ROC surfaces that captured the essential characteristics of the nonparametric estimates. Parameter estimates from model-based and empirical approaches were similar, likely due to the large sample size and appropriateness of the normality assumption for this data.

Limitations and Future Work

  • Visualization: The authors note that visualizing ROC surfaces becomes challenging for classification dimensions higher than three, suggesting a need for more insightful constructions for high-dimensional diagnostic accuracy studies.
  • Semiparametric Assumptions: While the paper focuses on normality for semiparametric models, it acknowledges that other parametric families (like Weibull or gamma) could be used, but warns about the impact of incorrect distributional assumptions.
  • Extension to M-classes: The paper briefly introduces the concept of high-dimensional ROC manifolds for M-category classification, suggesting this as a direction for future research, particularly for developing better visualization and interpretation tools.
  • Hypothesis Testing: The authors mention that hypothesis tests to compare the accuracy of two diagnostic tests for three classes can be naturally constructed from their methods, indicating a potential area for further development.

Summary updated

BibTeX
@article{li2009nonparametric,
  author = {Jialiang Li and Xiao‐Hua Zhou},
  title = {{Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface}},
  journal = {Journal of Statistical Planning and Inference},
  year = {2009},
  doi = {10.1016/j.jspi.2009.05.043},
}
Figure