Abstract
Explainability (XAI) is often viewed as a binary requirement in healthcare AI, yet clinical tasks vary significantly in complexity and risk. This paper proposes a systematic framework to determine the "Explanation Necessity" for medical AI models. By quantifying expert disagreement, protocol robustness, and data dimensionality, the authors move away from a one-size-fits-all approach toward a tiered system. This framework prioritizes transparency where it is needed most—such as in new-pattern discovery—while streamlining routine tasks like organ segmentation.
Introduction: Beyond the Black Box
In healthcare AI, the demand for explainability is frequently treated as a static requirement. However, the medical field encompasses a wide spectrum of tasks, from routine organ segmentation on CT scans to the experimental discovery of biomarkers for rare diseases. The authors argue that current literature lacks practical guidance on when a model requires a local explanation (specific to one patient) versus a global explanation (the model's overall logic). Without a structured framework, developers risk over-engineering explanations for simple tasks or providing insufficient transparency for high-stakes clinical decisions.
Methodology: The Two-Flow Decision Process
The researchers introduce a novel categorization system that sorts AI applications into four levels of Explanation Necessity. To assign an application to a category, the framework employs a two-flow decision process:
- Flow 1 (Inter-observer variability): This flow measures human expert agreement using metrics such as the Kappa statistic ($\kappa$) or Dice Similarity Coefficient (DSC). High disagreement among experts indicates that the AI's "reasoning" is more critical to investigate.
- Flow 2 (Representation dimensionality): This evaluates data complexity. Low-dimensional data (e.g., single-modality X-rays) generally requires less explanation than high-dimensional, multi-modal, or longitudinal data.
If these flows suggest different levels of XAI, a human "adjudicating expert" makes the final determination.
Categorizing Explanation Necessity
The framework defines four distinct tiers of transparency:
- Self-explainable: Tasks with minimal expert disagreement and robust protocols (e.g., basic organ segmentation). The output is directly understandable, making extra XAI unnecessary.
- Semi-explainable: Requires local explanations to support training and validation (e.g., Alzheimer’s classification from MRI).
- Non-explainable: Requires both local and global explanations due to high expert variability and weaker protocols.
- New-patterns discovery: Demands the highest level of XAI, aligning global explanations with ground-truth statistical models to validate novel clinical markers.
Quantitative Thresholds and Key Results
The paper establishes specific quantitative thresholds for "Initial Explanation Necessity" based on the probability of uncertainty ($1-\kappa$ or $1-DSC$).
A critical insight is the "Agreement Threshold": if inter-rater agreement is below 0.60, the task protocol itself may be flawed, and the AI should not be accepted for medical use regardless of its explainability. For accepted models, the thresholds are:
- 0.00–0.10: Self-explainable
- 0.11–0.20: Semi-explainable
- 0.21–0.30: Non-explainable
- 0.31–0.40: New-pattern discovery
Discussion: XAI as an Optimization and Validation Tool
The framework shifts the role of XAI from a mere trust-building exercise to a functional tool. For "self-explainable" tasks, XAI helps developers optimize performance. In "new-pattern discovery," it serves a validation role, ensuring the model's global logic aligns with statistical ground truth rather than relying on data artifacts. This formalization of global explanation ($g_z$) as a dimensionality reduction of local explanations provides a mathematical path to clinical acceptance.
Limitations and Future Work
While promising, the framework's current thresholds are primarily derived from medical imaging. Future research is needed to adapt these metrics for natural language processing or other healthcare domains. Additionally, the reliance on multiple experts for inter-rater studies may pose logistical challenges in niche fields. Finally, the definition of "high dimensionality" remains somewhat qualitative and requires further rigorous parameterization.
Summary updated