Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review

Felipe O. Giuste, Wenqi Shi, Yuanda Zhu, Tarun Naren, Monica Isgut, Ying Sha, Li Tong, Mitali S. Gupte, May D. WangIEEE Reviews in Biomedical Engineering 2021 · DOI · OpenReview

TL;DR

This systematic review examines how Explainable Artificial Intelligence (XAI) can bridge the gap between promising AI solutions for COVID-19 and their limited real-world clinical impact. By making AI models more transparent, XAI can foster trust, improve decision-making, and ultimately enhance the effectiveness of AI in healthcare, especially during pandemics.

What Problem Does This Paper Address?

The COVID-19 pandemic saw an explosion of AI-based solutions aimed at diagnosis, risk assessment, and patient management. However, despite impressive performance on benchmarks, very few of these AI tools have translated into significant clinical impact. The primary barrier identified is the "black box" nature of many AI models, which lack transparency and interpretability. Clinicians are hesitant to trust and adopt AI systems they don't understand, especially when dealing with critical decisions like patient diagnosis or treatment. This lack of interpretability also hinders researchers from identifying model flaws and improving performance. This paper addresses this critical gap by systematically reviewing how XAI techniques can make AI models more understandable and trustworthy for clinical applications, particularly in the context of pandemics.

Methods

This paper presents a systematic review of XAI methods applied to COVID-19 challenges. The authors followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for paper selection. They categorized XAI approaches into four main areas:

  • Data Augmentation: Using techniques like Generative Adversarial Networks (GANs) to create synthetic data for training, with XAI used to understand the latent space of GANs and ensure generated images have desired properties.
  • Outcome Prediction: Employing methods like perturbation-based (e.g., feature occlusion, SHAP), activation-based (e.g., CAM), gradient-based (e.g., saliency maps), mixed-based (e.g., Grad-CAM), and attention-based approaches to explain model predictions for diagnosis and risk assessment.
  • Unsupervised Clustering: Utilizing XAI to interpret patterns discovered in unlabeled data, helping to identify sub-phenotypes or novel biomarkers.
  • Image Segmentation: Applying XAI to understand how segmentation models delineate regions of interest in medical images, aiming to improve accuracy and trust.

The review also discusses the evolution of XAI from classical to modern and future trends, emphasizing the importance of both qualitative and quantitative evaluation of XAI methods.

Datasets

The paper references numerous publicly available datasets used in the reviewed studies, including:

  • Imaging Datasets: Coronacases.ORG, COVID-CT (UCSD), COVID-19 CT Lung and Infection Segmentation Dataset, COVID-19 Segmentation Dataset, MosMed COVID-19 Chest CT database, Società Italiana di Radiologia Medica (SIRM), AlforCOVID, British Society or Thoracic Imaging (BSTI), COVID-19 Image Collection, BIMCV-COVID19+, Radiopaedia COVID-19, Eurorad.
  • Non-Imaging Datasets: UCSF COVID-19 Research Data Mart, All of Us: COVID-19 research initiative, The 4CE Consortium, 2020 HRS COVID-19 Project, European CDC COVID-19 Data Tracker, US CDC COVID-19 Data Tracker.

The paper does not introduce new datasets but focuses on analyzing how existing datasets were used with XAI techniques.

Key Insights and Ideas

  • XAI is Crucial for Clinical Impact: The core message is that AI solutions for healthcare, especially during critical events like pandemics, will only achieve significant clinical impact if they are explainable. Transparency builds trust, which is essential for adoption.
  • Beyond Accuracy: Trust and Action: The ultimate goal of AI in clinical settings is not just high performance, but also instilling user trust and driving appropriate clinical action. XAI directly addresses the "user trust" component.
  • Evolution of XAI: The review highlights a shift from model-specific XAI methods to more general, model-agnostic approaches. There's also a trend towards combining global and local explanations for a more comprehensive understanding.
  • GANs and Interpretability: XAI is becoming vital for understanding the latent spaces of GANs used for data augmentation, allowing for better control over synthetic image generation and identification of desired properties.
  • Challenges in Evaluation: Quantitatively evaluating XAI methods is still an emerging area, with ongoing research into metrics that can objectively assess explanation quality and robustness.

Key Results

The paper is a systematic review, so it doesn't present single quantitative results like accuracy scores. Instead, it synthesizes findings from numerous studies. Key qualitative findings include:

  • XAI techniques can improve model performance by helping identify model flaws and potential biases.
  • XAI can validate extracted features, confirm heuristics, identify subgroups, and generate novel biomarkers.
  • Visualizations like heatmaps and saliency maps are widely used to highlight important regions in medical images, aiding clinician understanding.
  • Decision tree-based approaches and feature importance scores (like SHAP) are prevalent for interpreting EHR data in risk prediction.
  • Interactive user interfaces are increasingly important for presenting XAI results to clinicians in an understandable format.

Limitations and Future Work

Limitations:

  • Quantitative Evaluation of XAI: The paper notes that quantitative evaluation of XAI methods is still in its infancy, particularly for complex biomedical data.
  • Adversarial Attacks: The vulnerability of AI models to adversarial attacks is a significant concern, and robust validation methods are needed.
  • Data Scarcity and Quality: Challenges like small sample sizes, poor data quality, and imbalanced classes persist, requiring solutions like data imputation and augmentation.
  • Generalizability: Many studies are validated on single-center datasets, limiting their generalizability. Robust external validation is crucial.

Future Work:

  • Generalizable and Transferable XAI: Developing XAI modules that can be applied across various AI tasks and models is a key future direction.
  • Combination of Global and Local Explanations: Integrating both global (model-wide) and local (instance-specific) explanations will provide a more complete understanding for decision-making.
  • Risk Stratification: With decreasing COVID-19 incidence, risk stratification will become more critical, and XAI will play a vital role in this.
  • Framework for AI Development: The authors propose their checklist and insights as a guide for designing AI development workflows, aiming to improve the clinical implementation of AI solutions.
  • Addressing Novel Pandemics: The insights gained from XAI in the COVID-19 context can inform strategies for future disease outbreaks.

Summary updated

Abstract

Despite the myriad peer-reviewed papers demonstrating novel Artificial Intelligence (AI)-based solutions to COVID-19 challenges during the pandemic, few have made significant clinical impact. The impact of artificial intelligence during the COVID-19 pandemic was greatly limited by lack of model transparency. This systematic review examines the use of Explainable Artificial Intelligence (XAI) during the pandemic and how its use could overcome barriers to real-world success. We find that successful use of XAI can improve model performance, instill trust in the end-user, and provide the value needed to affect user decision-making. We introduce the reader to common XAI techniques, their utility, and specific examples of their application. Evaluation of XAI results is also discussed as an important step to maximize the value of AI-based clinical decision support systems. We illustrate the classical, modern, and potential future trends of XAI to elucidate the evolution of novel XAI techniques. Finally, we provide a checklist of suggestions during the experimental design process supported by recent publications. Common challenges during the implementation of AI solutions are also addressed with specific examples of potential solutions. We hope this review may serve as a guide to improve the clinical impact of future AI-based solutions.
BibTeX
@article{giuste2021explainable,
  author = {Felipe O. Giuste and Wenqi Shi and Yuanda Zhu and Tarun Naren and Monica Isgut and Ying Sha and Li Tong and Mitali S. Gupte and May D. Wang},
  title = {{Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review}},
  journal = {IEEE Reviews in Biomedical Engineering},
  year = {2021},
  doi = {10.1109/rbme.2022.3185953},
}
Figure