Comparison of Prediction Models for Heart Disease Data: Logistic Regression, Random Forest and Extreme Gradient Boosting

Jiaming TangHighlights 2025 · DOI

TL;DR

This paper evaluates three popular machine learning models—Logistic Regression (LR), Random Forest (RF), and Extreme Gradient Boosting (XGBoost)—to predict cardiovascular disease risk. By combining statistical filtering with Principal Component Analysis (PCA), the study finds that XGBoost achieves the highest accuracy (85.19%), outperforming traditional linear methods and ensemble alternatives.

What Problem Does This Paper Address?

Cardiovascular diseases (CVDs) remain the leading cause of death globally. While traditional clinical assessments rely on standard risk factors like age, blood pressure, and cholesterol, these linear approaches often struggle to capture the complex, non-linear interactions between biological variables.

The author tackles the challenge of model selection in a clinical context: which algorithm provides the best balance of predictive power and reliability when dealing with multi-dimensional patient data? The goal is to move beyond simple statistical correlations toward a robust predictive framework that can assist in early diagnosis and personalized medical intervention.

Methods

The research follows a structured machine learning pipeline:

  1. Data Preprocessing: Binary variables were encoded, and numerical features were normalized by their maximum values to ensure a comparable scale.
  2. Feature Selection & Filtering: The author used a two-step statistical filter. First, Variance Inflation Factor (VIF) was used to check for multicollinearity (ensuring variables weren't too highly correlated with each other). Second, P-value testing was applied to determine statistical significance; interestingly, Cholesterol and Fasting Blood Sugar (FBS) were excluded at this stage for having P-values > 0.05.
  3. Dimensionality Reduction: Principal Component Analysis (PCA) was employed to simplify the feature space. The author selected the first 7 principal components, which accounted for 85% of the total variance in the dataset.
  4. Model Training: Three distinct architectures were compared:
  5. Logistic Regression (LR): A baseline linear model using maximum likelihood estimation.
  6. Random Forest (RF): An ensemble of decision trees tuned to minimize Root Mean Square Error (RMSE).
  7. XGBoost: A gradient boosting framework optimized via stepwise grid search and stratified k-fold validation.

Datasets

The study utilized the Heart Disease Dataset from the UCI Machine Learning Repository (accessed via Kaggle).

  • Size: 270 observations with 13 initial variables.
  • Features: Includes clinical metrics like age, sex, chest pain type, blood pressure, EKG results, and Thallium stress test outcomes.
  • Split: A standard 80/20 ratio for training and testing.

Key Insights and Ideas

  • The Power of Boosting: XGBoost’s superior performance is attributed to its ability to capture complex, non-linear relationships that Logistic Regression misses. Its use of second-order gradients (Hessians) and built-in regularization helps it find a more nuanced "fit" for clinical data.
  • Statistical Counter-Intuition: One of the more surprising "aha" moments is the exclusion of Cholesterol and Fasting Blood Sugar based on P-values. While these are clinically vital indicators, in this specific dataset, they did not provide statistically significant predictive power compared to other features like "Number of major vessels" or "Thallium stress test" results.
  • Efficiency through PCA: The study demonstrates that you don't need all 13 variables to maintain high accuracy. Reducing the feature set to 7 principal components simplifies the model without a significant loss of information, which is crucial for building interpretable and efficient clinical tools.

Key Results

The models were evaluated primarily on classification accuracy derived from confusion matrices:

  • XGBoost: The clear winner with a test accuracy of 0.8519 (85.19%). It correctly identified 24 out of 30 positive cases in the test set.
  • Logistic Regression: Performed respectably as a baseline with an accuracy of 0.8333 (83.33%).
  • Random Forest: Surprisingly, the RF model reported a very low test accuracy of 0.1667. Looking at the confusion matrix, the model appears to have struggled significantly with false negatives, suggesting that the chosen parameters (ntree=100, mtry=6) or the PCA-transformed features may not have been optimal for this specific ensemble configuration.

Limitations and Future Work

The author identifies several critical areas for improvement:

  • Dataset Size and Diversity: With only 270 samples, the dataset is relatively small. The results may not generalize to broader, more diverse populations. The data was also skewed toward older male participants, leaving a gap in performance data for younger patients and women.
  • The "Black Box" Problem: While XGBoost is accurate, it lacks the inherent interpretability of Logistic Regression. Future work should focus on "clinical interpretability"—ensuring doctors can understand why a model flagged a patient as high-risk.
  • Model Architecture: The author suggests exploring deep neural networks or hybrid ensemble frameworks to further push the boundaries of predictive accuracy in future iterations.

Summary updated

Abstract

This paper aims to use predictive models such as the Logistic Regression (LR), Random Forest (RF) and Extreme Gradient Boosting (XGBoost) to predict the onset of cardiovascular disease based on relevant risk factors and make a comparison of the performance of these three models. First, this paper examined the statistical significance of the initial variables and the linear correlations between individual variables. To prepare the data for modeling, this paper first normalized all numerical and binary variables to ensure they were on a comparable scale. Dimensionality was then reduced using Principal Component Analysis (PCA), with the goal of simplifying the feature space while retaining meaningful variance. For the Random Forest (RF) model, parameter tuning prioritized minimizing RMSE, though the selected configuration also led to strong classification accuracy. XGBoost was optimized through a stepwise grid search combined with stratified k-fold validation, helping to maintain consistent performance across data splits. When comparing Logistic Regression, RF, and XGBoost, the latter consistently achieved the best classification results-likely due to its ability to capture complex and non-linear feature relationships. While these early findings are encouraging, especially for XGBoost, further validation on broader and more diverse datasets will be important before drawing firm conclusions about its clinical utility.
BibTeX
@inproceedings{tang2025comparison,
  author = {Jiaming Tang},
  title = {{Comparison of Prediction Models for Heart Disease Data: Logistic Regression, Random Forest and Extreme Gradient Boosting}},
  booktitle = {Highlights},
  year = {2025},
  doi = {10.54097/mzb6xv06},
}
Figure