On the Importance of Diversity in Re-Sampling for Imbalanced Data and Rare Events in Mortality Risk Models

Yuxuan Yang, Hadi Akbarzadeh Khorshidi, Uwe Aickelin, Aditi Nevgi, Elif Ekinci

Abstract

Surgical risk increases significantly when patients present with comorbid conditions. This has resulted in the creation of numerous risk stratification tools with the objective of formulating associated surgical risk to assist both surgeons and patients in decision-making. The Surgical Outcome Risk Tool (SORT) is one of the tools developed to predict mortality risk throughout the entire perioperative period for major elective in-patient surgeries in the UK. In this study, we enhance the original SORT prediction model (UK SORT) by addressing the class imbalance within the dataset. Our proposed method investigates the application of diversity-based selection on top of common re-sampling techniques to enhance the classifier's capability in detecting minority (mortality) events. Diversity amongst training datasets is an essential factor in ensuring re-sampled data keeps an accurate depiction of the minority/majority class region, thereby solving the generalization problem of mainstream sampling approaches. We incorporate the use of the Solow-Polasky measure as a drop-in functionality to evaluate diversity, with the addition of greedy algorithms to identify and discard subsets that share the most similarity. Additionally, through empirical experiments, we prove that the performance of the classifier trained over diversity-based dataset outperforms the original classifier over ten external datasets. Our diversity-based re-sampling method elevates the performance of the UK SORT algorithm by 1.4%.

Introduction

In clinical predictive modeling, the events of greatest interest—such as post-operative mortality—are often the rarest. In many surgical datasets, mortality rates can be as low as 3%, creating a significant "class imbalance." Standard machine learning algorithms typically aim to maximize overall accuracy, which often leads them to overlook the minority class entirely.

While common re-sampling techniques like Synthetic Minority Over-sampling Technique (SMOTE) or Random Over-Sampling (ROS) are used to balance these datasets, they frequently introduce redundant or "noisy" samples. This can blur the decision boundaries between classes and lead to poor model generalization. This research posits that the quality of a training set depends not just on the quantity of samples, but on their diversity.

Methodology

We propose a "drop-in" diversity-based selection functionality that can be integrated with any standard re-sampling technique (ROS, SMOTE, or Random Under-Sampling). The core of this approach is the Solow-Polasky measure, a mathematical framework used to quantify the diversity of a set of points based on their relative distances.

To manage the computational complexity of calculating diversity across large subsets, we implemented a greedy algorithm. The process involves:

  1. Starting with the full set of re-sampled data.
  2. Calculating the diversity contribution of each individual data point.
  3. Identifying and discarding points that add the least unique information using an optimized matrix-inversion formula.
  4. Repeating the process until the target dataset size is achieved.

This method ensures that the model learns from the most distinct examples, effectively defining the boundaries of the minority class without the interference of redundant data.

Experimental Setup

The proposed method was validated using two primary data sources:

  • The Austin Dataset: A real-world clinical dataset of 1,853 surgical inpatient records from Austin Health, Melbourne, including variables like age, sex, and the Charlson Comorbidity Index (CCI).
  • UCI Machine Learning Repository: Ten diverse benchmark datasets with varying degrees of class imbalance were used to demonstrate the method's generalizability across different domains.
Summary of the ten benchmark datasets (D1-D10) from the UCI repository used to evaluate the diversity-based re-sampling approach.

Results and Discussion

The empirical results demonstrate that diversity-based selection acts as a force multiplier, particularly in cases of extreme imbalance. A key finding of this study is that Precision-Recall AUC (PR-AUC) provides a more accurate assessment of performance for rare events than the more common ROC-AUC, which can appear deceptively optimistic even when a model fails to identify the minority class.

Performance Gains

Applying diversity-based re-sampling to the UK SORT algorithm resulted in a 1.4% improvement in predictive performance on the Austin dataset. Across the benchmark datasets, diversity-based methods significantly outperformed standard benchmarks in 54 out of 90 paired comparisons.

Comparative performance analysis across datasets, highlighting the consistent improvements achieved by diversity-based methods, particularly D-RUS.

As shown in the comparative analysis, Diversity-based Under-Sampling (D-RUS) emerged as a top performer, exceeding its non-diversity counterpart in 11 out of 12 test cases. This indicates that when under-sampling is necessary, selecting which data to retain based on diversity is significantly more effective than random selection.

Conclusion

This study highlights the critical role of data diversity in training robust models for rare event prediction. By incorporating the Solow-Polasky measure and a greedy selection algorithm, we successfully enhanced the UK SORT algorithm and demonstrated superior performance across multiple imbalanced datasets. Future work will explore alternative distance metrics, such as Mahalanobis Distance, and probability density-based approaches to further refine the measurement of sample diversity.

Summary updated

Abstract

Surgical risk increases significantly when patients present with comorbid conditions. This has resulted in the creation of numerous risk stratification tools with the objective of formulating associated surgical risk to assist both surgeons and patients in decision-making. The Surgical Outcome Risk Tool (SORT) is one of the tools developed to predict mortality risk throughout the entire perioperative period for major elective in-patient surgeries in the UK. In this study, we enhance the original SORT prediction model (UK SORT) by addressing the class imbalance within the dataset. Our proposed method investigates the application of diversity-based selection on top of common re-sampling techniques to enhance the classifier's capability in detecting minority (mortality) events. Diversity amongst training datasets is an essential factor in ensuring re-sampled data keeps an accurate depiction of the minority/majority class region, thereby solving the generalization problem of mainstream sampling approaches. We incorporate the use of the Solow-Polasky measure as a drop-in functionality to evaluate diversity, with the addition of greedy algorithms to identify and discard subsets that share the most similarity. Additionally, through empirical experiments, we prove that the performance of the classifier trained over diversity-based dataset outperforms the original classifier over ten external datasets. Our diversity-based re-sampling method elevates the performance of the UK SORT algorithm by 1.4$.
BibTeX
@inproceedings{yang2021importance,
  author = {Yuxuan Yang and Hadi Akbarzadeh Khorshidi and Uwe Aickelin and Aditi Nevgi and Elif Ekinci},
  title = {{On the Importance of Diversity in Re-Sampling for Imbalanced Data and Rare Events in Mortality Risk Models}},
  booktitle = {ACSW},
  year = {2021},
  url = {https://openreview.net/forum?id=rodV9AK96u},
}
Figure