Performance discrepancy mitigation in heart disease prediction for multisensory inter-datasets

Mahmudul Hasan, Md Abdus Sahid, Md Palash Uddin, Md Abu Marjan, Seifedine Kadry, Jungeun KimPeerJ Comput. Sci 2024 · DOI

TL;DR

This paper tackles the "inter-dataset discrepancy problem" in heart disease prediction, where models trained on one dataset perform poorly on another. They propose a preprocessing pipeline involving feature extraction (PCA) and feature selection (RF) to harmonize data from different sources, significantly improving model performance and enabling more robust, generalizable heart disease prediction systems.

What Problem Does This Paper Address?

Heart disease remains a leading cause of death globally, making accurate early prediction crucial. While machine learning (ML) offers powerful tools for this, most existing research trains and tests models on a single dataset (intra-dataset). This approach fails to account for the "inter-dataset discrepancy problem": models trained on one dataset often exhibit a significant drop in performance when applied to data from a different source, even if the underlying task is the same. This discrepancy arises from variations in data collection, feature distributions, and inherent biases across datasets. Developing a single, robust ML model that can generalize across diverse heart disease datasets is essential for creating more reliable and widely applicable diagnostic tools.

Methods

The authors propose a multi-stage preprocessing pipeline to mitigate the inter-dataset discrepancy. This pipeline consists of:

  • Missing Value Handling: Employing a Random Forest (RF) regression model to impute missing values, leveraging its robustness.
  • Log Transformation: Applied to normalize data distributions.
  • Outlier Detection and Removal: Using Turkey fences to identify and handle outliers.
  • Normalization: Min-Max scaling is used to bring all numerical features to a [0, 1] range.
  • Imbalance Data Handling: SMOTETomek ensemble methods are used to address class imbalance by oversampling the minority class and undersampling the majority class.
  • Dimensionality Reduction:
    • Feature Extraction: Principal Component Analysis (PCA) is used to reduce the dimensionality of the data by creating a smaller set of uncorrelated principal components.
  • Feature Selection: Random Forest (RF) is employed to select the most relevant features, aiming to improve model quality by removing less important ones.

These preprocessing steps are applied to individual datasets and various combinations of datasets before feeding them into several standard ML classifiers (SVM, KNN, DT, RF, XGBoost, GNB, LR, MLP).

Datasets

The study utilizes five publicly available heart disease datasets:

  • Cleveland
  • Hungarian
  • Switzerland
  • Long Beach VA
  • StatLog

These datasets were chosen for their feature similarity and common use in existing research. A combined dataset was also formed by merging these five datasets after removing duplicates.

Key Insights and Ideas

  • Inter-dataset discrepancy is a significant hurdle: The study clearly demonstrates that models trained on one heart disease dataset perform poorly on others without specific mitigation strategies.
  • Preprocessing is key: A well-designed preprocessing pipeline, rather than complex model architectures, can effectively address inter-dataset discrepancies.
  • RF excels in feature selection: Random Forest proved to be a highly effective technique for selecting the most informative features, leading to improved model performance.
  • PCA offers good feature extraction: While RF showed a slight edge, PCA also significantly improved performance by reducing data dimensionality.
  • Combined approach is powerful: The proposed pipeline, integrating multiple preprocessing steps, significantly enhances classifier performance across various dataset combinations.

Key Results

  • Without the proposed preprocessing pipeline, classifier performance was often unstable and significantly lower when tested on different datasets, highlighting the inter-dataset discrepancy.
  • After applying the PCA-based preprocessing pipeline, classifier performance improved considerably, with some algorithms achieving up to 100% accuracy, precision, recall, and F1-score, and 100% specificity and AUC.
  • The RF-based feature selection pipeline further boosted performance, with RF itself showing superior results in predicting heart disease, achieving up to 96% accuracy, precision, recall, and 99% AUC score in inter-dataset setups.
  • The study shows that the proposed pipeline mitigates the inter-dataset performance discrepancy problem, leading to more stable and higher classifier performance across different dataset combinations.

Limitations and Future Work

  • Dataset Similarity: The proposed method is applied to secondary datasets with similar features. The authors suggest exploring this domain more for datasets with dynamic updates and feature differences.
  • Generalizability: While the study focuses on heart disease, the authors propose that the preprocessing pipeline could be extended to minimize similar issues for other chronic diseases, aiding in the creation of global models and datasets.
  • Data Confidentiality: The authors note that traditional ML methods cannot ensure data confidentiality and suggest future studies focus on privacy-preserving distributed computing.

Summary updated

BibTeX
@article{hasan2024performance,
  author = {Mahmudul Hasan and Md Abdus Sahid and Md Palash Uddin and Md Abu Marjan and Seifedine Kadry and Jungeun Kim},
  title = {{Performance discrepancy mitigation in heart disease prediction for multisensory inter-datasets}},
  journal = {PeerJ Comput. Sci.},
  year = {2024},
  doi = {10.7717/peerj-cs.1917},
}
Figure