Guidelines and evaluation for clinical explainable AI on medical image analysis

Weina Jin, Xiaoxiao Li, Mostafa Fatehi, Ghassan Hamarneh

Introduction

In medical imaging, AI often operates as a "black box," providing diagnoses without transparent reasoning. While numerous Explainable AI (XAI) techniques exist, they are frequently evaluated on visual "plausibility" to humans rather than their actual impact on clinical decision-making. Jin et al. address this gap by introducing the Clinical XAI Guidelines, a framework designed to ensure AI explanations are technically sound and clinically viable.

The Clinical XAI Framework

The authors propose five guidelines (G1–G5) categorized into Clinical Usability, Evaluation, and Operation. These guidelines shift the focus from mere visual appeal to technical truthfulness and practical utility in a medical environment.

Table 1: Detailed definitions and assessment methods for the five Clinical XAI Guidelines (G1-G5).
  • Clinical Usability (G1 & G2): Focuses on whether explanations are understandable to clinicians without technical backgrounds (G1) and relevant to their diagnostic reasoning (G2).
  • Evaluation (G3 & G4): Ensures explanations truthfully reflect the model's internal logic (G3) and help clinicians identify potential AI errors or biases (G4).
  • Operation (G5): Requires that the computational speed of generating explanations is compatible with real-time clinical workflows.

Evaluation Methodology

The study evaluated 16 popular post-hoc XAI algorithms, such as Grad-CAM and LIME, across two distinct clinical tasks:

  1. Glioma Grading (Brain MRI): Utilizing the BraTS 2020 dataset and a synthetic variant to establish a "ground truth" for feature importance.
  2. Knee Lesion Identification (Knee MRI): Using the MRNet dataset to identify meniscus tears.

A key contribution is the Modality-Specific Feature Importance (MSFI) metric, which assesses how well XAI methods identify the most influential image modality (e.g., T1 vs. FLAIR) in multi-modal diagnoses—a critical aspect of radiologic reasoning.

Key Results and Insights

The evaluation revealed a sobering reality: most current XAI methods are not yet suitable for high-stakes clinical use. While they generally succeed in being understandable and efficient, they fail in the most critical areas of truthfulness and error detection.

Figure 1: High-level summary of the Clinical XAI Guidelines and the evaluation performance of 16 heatmap-based methods.
  • The Plausibility Trap: Many explanations look "right" to a human (e.g., highlighting a tumor) but do not actually reflect the AI's internal logic (G3). The authors found that most methods performed no better than a random baseline at identifying truly important features in the knee MRI task.
  • Inability to Spot Errors: Current heatmaps are generally poor at helping users distinguish between a correct AI prediction and an incorrect one (G4). This can lead to dangerous over-trust in a faulty model.
  • Efficiency Trade-offs: Gradient-based methods are nearly instant, but perturbation-based methods can take up to 30 minutes per image, making them impractical for a busy clinic (G5).

Conclusion

The study highlights that "pretty" heatmaps are insufficient for clinical practice. Truthfulness (G3) must be the prerequisite for any clinical XAI tool. Future work should move beyond simple heatmaps toward more robust explanation forms, such as example-based or rule-based methods, to build genuine, evidence-based trust in medical AI.

Summary updated

BibTeX
@article{jin2023guidelines,
  author = {Weina Jin and Xiaoxiao Li and Mostafa Fatehi and Ghassan Hamarneh},
  title = {{Guidelines and evaluation for clinical explainable AI on medical image analysis}},
  journal = {Medical Image Analysis},
  year = {2023},
  doi = {10.1016/j.media.2022.102684},
}
Figure