A Resampling Method to Improve the Prognostic Model of End-Stage Kidney Disease: A Better Strategy for Imbalanced Data

Xi Shi, Tingyu Qu, Gijs Van Pottelbergh, Marjan van den Akker, Bart De MoorFrontiers in Medicine 2022 · DOI

TL;DR

This paper tackles the challenge of predicting End-Stage Kidney Disease (ESKD) in patients with Chronic Kidney Disease (CKD), a problem plagued by imbalanced data where ESKD cases are rare. They introduce a resampling technique, SMOTE-ENN, to preprocess the data, significantly improving the identification of high-risk patients compared to traditional models, albeit with a higher rate of misclassifying non-ESKD patients.

What Problem Does This Paper Address?

Predicting the progression of Chronic Kidney Disease (CKD) to End-Stage Kidney Disease (ESKD) is crucial for timely intervention and better patient outcomes. However, a significant challenge in this area is the imbalanced nature of the data. In CKD patient populations, only a small fraction will actually progress to ESKD, while the vast majority will not. This disparity creates a problem for predictive models, particularly those relying on metrics like accuracy. Standard models often achieve high accuracy by simply predicting that no one will develop ESKD, effectively ignoring the minority class (ESKD patients) and failing to identify those truly at risk. Previous studies have often overlooked the impact of this imbalanced data structure on model performance and the adequacy of evaluation metrics.

Methods

The core innovation in this paper is the application of a resampling method as a preprocessing step to address the imbalanced data. Specifically, they employed the Synthetic Minority Over-Sampling Technique-Edited Nearest Neighbor (SMOTE-ENN).

  • SMOTE (Synthetic Minority Over-Sampling Technique): This is an oversampling technique that generates synthetic data points for the minority class (ESKD patients) by interpolating between existing minority class instances and their nearest neighbors. This helps to balance the class distribution without simply duplicating existing data.
  • ENN (Edited Nearest Neighbors): This is an undersampling technique that removes instances from the majority class (non-ESKD patients) that are misclassified by their nearest neighbors. This helps to clean up the majority class and remove potentially noisy or ambiguous data points.

SMOTE-ENN combines these two approaches. The authors tested various resampling methods but found SMOTE-ENN to be the most effective for their dataset. This preprocessed data was then used to train a logistic regression model. The performance of this SMOTE-ENN enhanced logistic regression model was then compared against a standard logistic regression model and a Cox proportional hazards model, both trained on the original, imbalanced data.

Datasets

The study utilized the Intego database, a longitudinal electronic health record database of patients in general practice in Flanders, Belgium. The data spanned from 2005 to 2015 and included approximately 300,000 individual patients. For this specific study, the researchers focused on patients with CKD who were older than 50 years, resulting in a dataset of 11,645 patients (11,424 with CKD and 221 who developed ESKD). The average observation interval was about 1.358 years, making it a roughly 1.5-year predictive model.

Key Insights and Ideas

  • The "Accuracy Paradox" is Real and Problematic: The paper strongly highlights how standard accuracy metrics can be highly misleading in imbalanced datasets. A model predicting "no ESKD" for everyone can achieve near-perfect accuracy, masking its complete failure to identify actual at-risk patients.
  • Resampling is a Powerful Preprocessing Tool: SMOTE-ENN effectively addresses the imbalanced data issue by creating a more balanced training set, allowing the logistic regression model to learn the patterns of the minority class (ESKD progression) more effectively.
  • Trade-offs are Inevitable: Improving the detection of the minority class (ESKD patients) comes at the cost of misclassifying more instances of the majority class (non-ESKD patients). This is a critical trade-off that needs to be understood and managed.
  • Beyond Standard Metrics: The study emphasizes the need for evaluation metrics that account for the differential costs of misclassification, leading them to favor the Fβ score (specifically F3) over simple accuracy or AUC.
  • Clinical Relevance of Improved Minority Class Detection: While misclassifying some non-ESKD patients, the improved identification of high-risk individuals has significant clinical implications for preventive care, which is often less burdensome than managing advanced ESKD.

Key Results

  • Improved ESKD Identification: The SMOTE-ENN + logistic regression model correctly identified 86.3% of patients with ESKD. This is a substantial improvement compared to the logistic regression (3.9%) and Cox proportional hazards models (2.0%) trained on the original data.
  • Higher F3 Score: The F3 score, a metric that balances precision and recall with a penalty for false negatives, was significantly higher for the SMOTE-ENN + logistic regression model (0.245) compared to the standard logistic regression (0.043) and Cox proportional hazards model (0.023).
  • Misclassification of Negative Cases: The improved ESKD identification came with a cost: 45.2% of non-ESKD patients were misclassified as high-risk by the SMOTE-ENN model.
  • Comparable AUC: While the accuracy and recall for ESKD detection were vastly different, the Area Under the Curve (AUC) for all three models remained relatively similar, underscoring the limitations of AUC in highly imbalanced scenarios.

Limitations and Future Work

  • Misclassification of Negative Cases: The authors acknowledge the significant limitation of misclassifying 45% of non-ESKD patients. While they argue this is clinically acceptable given the benefits of early intervention, it remains a key area for improvement.
  • Data Limitations: The study was limited by the availability of certain data points (e.g., proteinuria) and the relatively short average observation period (~1.5 years). Longer follow-up periods could potentially reveal more ESKD progressions.
  • Generalizability: While the technique is presented as broadly applicable to other chronic diseases with imbalanced data, further validation on different datasets and disease contexts is needed.
  • Future Work: The authors suggest exploring different resampling techniques, other prognostic models beyond logistic regression, and further investigation into the optimal trade-off between detecting true positives and minimizing false positives. They also propose that nephrologists could collect more specific data to improve diagnostic accuracy.

Summary updated

Abstract

This study pointed out the imbalanced data structure and its effects on prediction accuracy, which were not thoroughly discussed in previous studies. We were able to identify patients with high risk for ESKD better from a clinical perspective by using the resampling method. But, it has the limitation of the high misclassification of negative cases. The technique can be widely used in other clinical topics when imbalanced data structure should be considered.
BibTeX
@inproceedings{shi2022resampling,
  author = {Xi Shi and Tingyu Qu and Gijs Van Pottelbergh and Marjan van den Akker and Bart De Moor},
  title = {{A Resampling Method to Improve the Prognostic Model of End-Stage Kidney Disease: A Better Strategy for Imbalanced Data}},
  booktitle = {Frontiers in Medicine},
  year = {2022},
  doi = {10.3389/fmed.2022.730748},
}
Figure