Evaluation of the accuracy of diagnostic scales for a syndrome in Chinese medicine in the absence of a gold standard

Xiao Nan Wang, Vanessa Zhou, Qiang Liu, Ying Gao, Xiao‐Hua ZhouChinese 2016 · DOI

TL;DR

This paper tackles the challenge of evaluating diagnostic tools in Traditional Chinese Medicine (TCM) where a definitive "gold standard" for syndrome diagnosis is often absent. By employing a Bayesian approach that accounts for imperfect expert consensus, the study demonstrates that a newly developed diagnostic scale (ISDS) for "internal wind" syndrome in stroke patients is more accurate than an older scale (SSDC).

What Problem Does This Paper Address?

Traditional Chinese Medicine (TCM) relies on the concept of "syndromes" (zhengs) to understand and treat illness. These syndromes are diagnosed holistically, integrating various clinical observations. However, this subjective nature means that different practitioners may arrive at different diagnoses for the same patient, making it difficult to objectively assess the accuracy of diagnostic tools. Traditionally, expert consensus has been used as a "gold standard," but this is problematic because expert opinions themselves are not always perfectly accurate and can introduce bias. This paper specifically addresses the challenge of evaluating diagnostic scales for "internal wind" syndrome in ischemic stroke patients without a perfect gold standard, aiming to provide a more reliable assessment of their accuracy.

Methods

The core of this paper's methodological contribution lies in its use of a Bayesian approach to estimate diagnostic test accuracy in the absence of a gold standard. This is crucial because traditional methods that rely on a gold standard can be biased if that standard is imperfect. The authors employ a latent class model that allows for conditional dependence between diagnostic tests, acknowledging that different scales might not err independently.

Specifically, they model the observed data (cross-tabulations of diagnoses from three sources: SSDC, ISDS, and Expert Syndrome Differentiation - ESD) and use Bayesian inference to estimate the parameters of interest: the prevalence of the syndrome, the sensitivity (Se), and the specificity (Sp) of each diagnostic test. The Bayesian method allows them to incorporate prior information (based on existing literature and expert opinion) and update it with the observed data to derive posterior distributions for these parameters. They then use the median of these posterior distributions as their estimates.

The paper also details the selection of prior distributions for the model parameters, using methods that incorporate existing knowledge about the likely performance of the diagnostic tests. Markov Chain Monte Carlo (MCMC) techniques (specifically using WinBUGS software) are used to computationally derive these posterior distributions, as direct analytical solutions are often intractable for complex models.

Datasets

The study utilized data from 204 participants diagnosed with ischemic stroke. These participants were assessed using three diagnostic methods:

  • Stroke Syndrome Differentiation Diagnostic Criterion (SSDC)
  • Ischemic Stroke TCM Syndrome Diagnostic Scale (ISDS)
  • Expert Syndrome Differentiation (ESD)

The data was collected over a 10-month period (February 2008 to November 2008) across 10 hospitals in nine Chinese cities. This dataset represents a real-world clinical scenario for evaluating diagnostic scales for a specific TCM syndrome.

Key Insights and Ideas

  • The "gold standard" problem is real and impactful: The paper highlights that relying on expert consensus as a gold standard can lead to biased estimates of diagnostic test accuracy, potentially under- or overestimating true performance.
  • Bayesian methods offer a robust solution: The Bayesian framework, particularly when accounting for conditional dependence, provides a powerful way to evaluate diagnostic tests when a perfect gold standard is unavailable.
  • TCM diagnostic scales can be improved: The study demonstrates that newer, standardized scales like the ISDS can outperform older ones, suggesting a path towards more objective and reliable TCM diagnostics.
  • Internal wind syndrome in stroke is complex: The results show that even expert diagnoses (ESD) are not perfect, underscoring the inherent complexity in diagnosing TCM syndromes.

Key Results

The study found that after adjusting for imperfect gold standard bias using the Bayesian method:

  • The ISDS (Ischemic Stroke TCM Syndrome Diagnostic Scale) demonstrated higher accuracy than the SSDC (Stroke Syndrome Differentiation Diagnostic Criterion) for diagnosing internal wind syndrome in ischemic stroke patients.
  • Specifically, the estimated sensitivity of the ISDS was 0.884 and its specificity was 0.875.
  • In comparison, the SSDC had an estimated sensitivity of 0.687 and a specificity of 0.776.
  • The expert syndrome differentiation (ESD) also showed high accuracy, with a sensitivity of 0.813 and a specificity of 0.922, but was not perfect.
  • The estimated prevalence rate of internal wind syndrome in the study population was approximately 0.648.

Limitations and Future Work

  • Prior distribution sensitivity: While the authors tested the sensitivity of their results to different prior distributions and found consistent results, the choice of priors can still influence the posterior estimates. Future work could explore more sophisticated methods for prior selection or sensitivity analyses.
  • Conditional dependence assumptions: The model assumes conditional dependence between the two diagnostic scales (SSDC and ISDS) but independence of the expert opinion (ESD) from these scales, given the true syndrome status. While justified, exploring alternative dependence structures could be a future direction.
  • Generalizability: The study focused on "internal wind" syndrome in ischemic stroke patients. Future research could apply this Bayesian methodology to evaluate diagnostic scales for other TCM syndromes and in different patient populations to assess generalizability.
  • Clinical utility: While accuracy is evaluated, further research could explore the clinical utility and cost-effectiveness of the ISDS in real-world TCM practice.

Summary updated

Abstract

After adjusting for imperfect gold standard bias, we found that both the sensitivity and specificity of the ISDS were higher than those of the SSDC for diagnosis of internal wind syndrome in ischemic stroke patients.
BibTeX
@inproceedings{wang2016evaluation,
  author = {Xiao Nan Wang and Vanessa Zhou and Qiang Liu and Ying Gao and Xiao‐Hua Zhou},
  title = {{Evaluation of the accuracy of diagnostic scales for a syndrome in Chinese medicine in the absence of a gold standard}},
  booktitle = {Chinese},
  year = {2016},
  doi = {10.1186/s13020-016-0100-2},
}
Figure