Unboxing Industry-Standard AI Models for Male Fertility Prediction with SHAP

Debasmita GhoshRoy, Parvez Alvi, Kc SantoshHealthcare 2023 · OpenReview

TL;DR

This paper tackles the "black box" problem in AI models for male fertility prediction by integrating Explainable AI (XAI) techniques, specifically SHAP, with standard machine learning models. By doing so, they not only achieve robust predictions but also provide crucial insights into why a model makes a certain prediction, enhancing trust and utility for clinicians.

What Problem Does This Paper Address?

Male infertility is a significant issue, affecting a substantial portion of couples worldwide, and male factors contribute to a large percentage of these cases. Despite its prevalence, male infertility is often underrecognized. Environmental and lifestyle factors are increasingly implicated in declining semen quality. While Artificial Intelligence (AI) and Machine Learning (ML) offer promising avenues for early detection and prediction of male fertility, many existing AI models operate as "black boxes." This lack of transparency makes it difficult for clinicians to trust and interpret their predictions, hindering their adoption in real-world clinical practice and limiting their potential for guiding treatment planning. This paper aims to bridge this gap by making these AI models more interpretable.

Methods

The researchers employed a comprehensive approach involving several key steps:

  • Industry-Standard ML Models: Seven well-established ML classifiers were utilized: Support Vector Machine (SVM), Random Forest (RF), Decision Tree (DT), Logistic Regression (LR), Naïve Bayes (NB), AdaBoost (ADA), and Multi-Layer Perceptron (MLP).
  • Data Balancing: To address class imbalance in the dataset (where normal samples significantly outnumbered altered samples), the Synthetic Minority Over-sampling Technique (SMOTE) was applied. This technique generates synthetic data points for the minority class to create a more balanced training set.
  • Validation Schemes: Two common cross-validation techniques were used to ensure model robustness and prevent overfitting: hold-out validation (80% train, 20% test) and k-fold cross-validation (specifically 5-fold CV).
  • Explainable AI (XAI) with SHAP: The core innovation is the application of Shapley Additive Explanations (SHAP). SHAP is a game theory-based approach that assigns an importance value to each feature for a particular prediction, explaining how each feature contributes to the model's output. This "unboxes" the black-box models, revealing the decision-making process.
  • Performance Metrics: Standard metrics like Accuracy (ACC), Sensitivity (SEN), Specificity (SPEC), F1-Score, and Area Under the Curve (AUC) were used to evaluate model performance.

Datasets

The study utilized a publicly available dataset from the UCI repository. This dataset contains 9 input features related to environmental and lifestyle factors, along with a target class indicating seminal quality.

  • Original Dataset: 100 samples, with 88 classified as normal seminal quality and 12 as altered seminal quality. This represents an imbalance ratio of approximately 1:7.
  • SMOTE-Augmented Dataset: After applying SMOTE, the training dataset size was increased by 600%, resulting in a more balanced distribution for model training.

Key Insights and Ideas

  • Interpretability is Key for Clinical Adoption: The study strongly emphasizes that high accuracy alone is insufficient for AI models in healthcare. Clinicians need to understand why a prediction is made to trust and act upon it. SHAP provides this crucial interpretability.
  • Feature Importance Varies by Model and Dataset: The SHAP analysis revealed that the most influential features for predicting male fertility can differ significantly between different AI models and whether the dataset is balanced or imbalanced. This highlights the need for model-specific explanations.
  • Age is a Dominant Predictor: Across several models and analyses, "Age at the time of analysis" (f2) consistently emerged as a highly significant feature influencing male fertility prediction.
  • Surgical Intervention's Low Impact: Conversely, "Surgical intervention" (f5) was often identified as one of the least significant features in many of the models, suggesting it has a minimal direct impact on the AI's prediction in this dataset.
  • RF and ADA Shine: Random Forest (RF) and AdaBoost (ADA) generally performed very well, achieving high accuracy and AUC scores, especially when combined with SMOTE and appropriate validation.

Key Results

  • Optimal Performance: The Random Forest (RF) model, when trained on a balanced dataset using 5-fold CV, achieved an optimal accuracy of 90.47% and an AUC of 0.998.
  • Good vs. Poor Performers:
    • Good Performers: ADA and RF consistently showed strong performance across various validation schemes and dataset types.
  • Poor Performers: DT, LR, and NB generally exhibited lower accuracy and AUC values, particularly on imbalanced datasets or with certain validation methods.
  • SHAP Insights: SHAP analysis provided detailed feature importance rankings for each model. For instance, in the RF model (balanced, 5-fold CV), 'Age' (f2) and 'Freq of alcohol consumption' (f7) were top features, while 'Surgical intervention' (f5) was least significant.

Limitations and Future Work

  • Limited Sample Size: The study acknowledges that the dataset, while from a reputable source, has a limited number of samples (100 initially). This can impact the generalizability of the findings.
  • Single Dataset: The analysis was conducted on a single dataset. Future work could involve testing these models and XAI approaches on larger, more diverse datasets.
  • Focus on Specific Features: The dataset primarily includes environmental and lifestyle factors, not direct semen analysis parameters. Future research could integrate these clinical parameters for a more holistic prediction.
  • Exploring Other AI Algorithms: The authors suggest exploring other AI algorithms and different sampling techniques for future model designs.
  • Real-time Implementation: While the XAI approach facilitates commercialization, further work on seamless real-time implementation is a potential future direction.

Summary updated

BibTeX
@inproceedings{ghoshroy2023unboxing,
  author = {Debasmita GhoshRoy and Parvez Alvi and KC Santosh},
  title = {{Unboxing Industry-Standard AI Models for Male Fertility Prediction with SHAP}},
  booktitle = {Healthcare},
  year = {2023},
  url = {https://openreview.net/forum?id=05wd3Lp8vW},
}
Figure