Evaluation of machine learning algorithms for predicting direct-acting antiviral treatment failure among patients with chronic hepatitis C infection

Haesuk Park, Wei‐Hsuan Lo‐Ciganic, James L. Huang, Yonghui Wu, Linda Henry, Joy Peter, Mark Sulkowski, David R. NelsonScientific 2022 · DOI

TL;DR

This paper leverages machine learning (ML) to predict treatment failure in patients with chronic Hepatitis C Virus (HCV) undergoing direct-acting antiviral (DAA) therapy. The developed ML models, particularly the Elastic Net (EN) algorithm, significantly outperform traditional logistic regression, identifying key risk factors and enabling better patient stratification for improved treatment outcomes.

What Problem Does This Paper Address?

Despite the high efficacy of Direct-Acting Antiviral (DAA) therapies for Hepatitis C Virus (HCV), a small but significant percentage of patients still experience treatment failure. This failure not only necessitates costly retreatment but also contributes to the ongoing burden of liver-related morbidity, transplantation, and mortality. Identifying patients at high risk of failure before or during treatment is crucial for optimizing treatment strategies and potentially intervening with modifiable factors. Previous attempts using machine learning had limitations, suggesting that incorporating during-treatment predictors could improve accuracy.

Methods

The researchers employed and compared four machine learning algorithms: Elastic Net (EN), Random Forest (RF), Gradient Boosting Machine (GBM), and Feedforward Neural Network (FNN). These models were trained and validated on a large dataset of HCV patients. Their performance was benchmarked against a traditional Multivariable Logistic Regression (MLR) model. Key evaluation metrics included the C-statistic (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and number needed to evaluate (NNE). The EN model was specifically highlighted for its parsimony, requiring the fewest predictors.

Datasets

The study utilized data from the HCV-TARGET registry, a longitudinal, prospective, observational cohort study of real-world DAA therapy administration. The dataset comprised 6525 HCV-infected adults who initiated all-oral DAA treatment between February 2014 and 2018, with available virologic outcome data. The data was split into a training set (n=4894) and a validation set (n=1631).

Key Insights and Ideas

  • ML Outperforms Traditional Methods: Machine learning models, especially EN, RF, and FNN, demonstrated significantly better predictive performance for DAA treatment failure compared to standard logistic regression. This highlights the power of ML in capturing complex relationships in clinical data.
  • Parsimony Matters: The Elastic Net (EN) model achieved comparable predictive accuracy to more complex ML models but used a substantially smaller set of predictors. This makes it more practical for clinical implementation and reduces computational burden.
  • During-Treatment Factors are Crucial: Incorporating predictors that emerge during treatment (e.g., treatment duration, discontinuation due to adverse events) alongside baseline characteristics significantly improved the prediction of DAA failure, addressing a limitation of prior studies.
  • Identifying Modifiable Risk Factors: The study identified several key predictors of treatment failure, including male sex, shorter treatment duration (<8 weeks), treatment discontinuation due to adverse events, low albumin, high bilirubin, advanced liver disease, and the use of tobacco, alcohol, or vitamins. This provides actionable targets for intervention.
  • Risk Stratification is Effective: The EN model effectively stratified patients into distinct risk subgroups, with a disproportionately high percentage of treatment failures concentrated in the top risk deciles. This allows for targeted monitoring and support.

Key Results

  • Predictive Accuracy: The ML models achieved C-statistics ranging from 0.72 to 0.75, significantly outperforming MLR (C-statistic = 0.51).
  • EN Model Performance: The EN model achieved a C-statistic of 0.74, with 58.4% sensitivity and 77.8% specificity when using the Youden index for threshold selection. It required only 9 patients to evaluate to identify 1 DAA treatment failure.
  • Risk Group Concentration: Over 60% of treatment failures were identified within the top three risk decile subgroups predicted by the EN model.
  • Top Predictors: Key predictors identified by the EN model included male sex, treatment duration < 8 weeks, treatment discontinuation due to adverse events, albumin < 3.5 g/dL, total bilirubin > 1.2 g/dL, advanced liver disease, and use of tobacco, alcohol, or vitamins.

Limitations and Future Work

The authors acknowledge several limitations:

  • No Baseline Resistance Data: The study did not include data on baseline resistance-associated substitutions (RASs), which can influence treatment outcomes. However, they note that routine RAS testing is not always recommended.
  • External Validation: While the study used a validation set from the same registry, external validation with independent datasets is recommended.
  • Causality vs. Association: The results represent associations, not causal inferences, between predictors and treatment failure.
  • Unknown Reasons for Discontinuation: The specific reasons for treatment discontinuation due to adverse events were not detailed, which could provide further insights.

Future work could focus on validating these models in diverse populations, exploring the causal mechanisms behind the identified modifiable risk factors, and integrating these ML tools into clinical decision support systems to proactively manage patients at risk of DAA treatment failure.

Summary updated

Abstract

Despite the availability of efficacious direct-acting antiviral (DAA) therapy, the number of people infected with hepatitis C virus (HCV) continues to rise, and HCV remains a leading cause of liver-related morbidity, liver transplantation, and mortality. We developed and validated machine learning (ML) algorithms to predict DAA treatment failure. Using the HCV-TARGET registry of adults who initiated all-oral DAA treatment, we developed elastic net (EN), random forest (RF), gradient boosting machine (GBM), and feedforward neural network (FNN) ML algorithms. Model performances were compared with multivariable logistic regression (MLR) by assessing C statistics and other prediction evaluation metrics. Among 6525 HCV-infected adults, 308 patients (4.7%) experienced DAA treatment failure. ML models performed similarly in predicting DAA treatment failure (C statistic [95% CI]: EN, 0.74 [0.69-0.79]; RF, 0.74 [0.69-0.80]; GBM, 0.72 [0.67-0.78]; FNN, 0.75 [0.70-0.80]), and all 4 outperformed MLR (C statistic [95% CI]: 0.51 [0.46-0.57]), and EN used the fewest predictors (n = 27). With Youden index, the EN had 58.4% sensitivity and 77.8% specificity, and nine patients were needed to evaluate to identify 1 DAA treatment failure. Over 60% treatment failure were classified in top three risk decile subgroups. EN-identified predictors included male sex, treatment < 8 weeks, treatment discontinuation due to adverse events, albumin level < 3.5 g/dL, total bilirubin level > 1.2 g/dL, advanced liver disease, and use of tobacco, alcohol, or vitamins. Addressing modifiable factors of DAA treatment failure may reduce the burden of retreatment. Machine learning algorithms have the potential to inform public health policies regarding curative treatment of HCV.
BibTeX
@inproceedings{park2022evaluation,
  author = {Haesuk Park and Wei‐Hsuan Lo‐Ciganic and James L. Huang and Yonghui Wu and Linda Henry and Joy Peter and Mark Sulkowski and David R. Nelson},
  title = {{Evaluation of machine learning algorithms for predicting direct-acting antiviral treatment failure among patients with chronic hepatitis C infection}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-22819-4},
}
Figure