TL;DR
This paper provides a historical and conceptual overview of explainable AI (XAI) research in the biomedical field, revealing a significant surge in publications around October 2020. The authors hypothesize that the COVID-19 pandemic acted as a catalyst, accelerating an existing trend towards XAI adoption to build trust and transparency in AI-driven healthcare.
What Problem Does This Paper Address?
The increasing integration of Artificial Intelligence (AI) in healthcare promises significant advancements, but the "black-box" nature of many AI models poses a major challenge. Clinicians, regulators, and patients need to understand why an AI makes a particular decision to trust and adopt these technologies, especially in high-stakes medical scenarios. This lack of transparency hinders the widespread adoption of AI in healthcare. While explainable AI (XAI) aims to address this, the field of biomedical XAI is rapidly evolving, and a clear understanding of its historical trajectory and current trends is needed.
Methods
The authors conducted a systematic review of biomedical XAI research using the PubMed database. They employed both manual and automatic methods to identify and label 1,276 peer-reviewed papers published between January 2010 and November 2022. To analyze trends, they fitted a change point detection model to publication dates and manually annotated the role of COVID-19 in each paper (not mentioned, used as data/use case, or mentioned as a driving force). They also categorized studies into five types: review/meta-analysis, discussion of XAI concepts, introduction of novel XAI methods, evaluation/application of XAI methods, and datasets/tools supporting XAI. Citation analysis was used to gauge public interest.
Datasets
The primary data source for this study was the PubMed database. No new datasets or benchmarks were introduced. The authors analyzed existing publications related to XAI in the biomedical domain.
Key Insights and Ideas
- COVID-19 as a Catalyst: The pandemic appears to have significantly accelerated the pace of biomedical XAI research, with a notable inflection point in publication rates around October 2020. This suggests that the urgent need for reliable AI solutions during the pandemic spurred greater interest in explainability.
- Focus on Application over Novelty: Current research efforts in biomedical XAI seem to prioritize explaining existing "black-box" models (post hoc explanations) over developing entirely new, inherently interpretable AI architectures. This is likely due to the relative ease of applying existing XAI techniques.
- Explainability vs. Trustworthiness: While XAI aims to build trust, the authors highlight that trustworthiness is a broader concept encompassing not just technology but also practices and policy.
- Causality's Role: The study acknowledges the importance of causality in XAI, particularly for understanding "why" a model makes a prediction and for exploring counterfactual scenarios. However, the growth in explainability publications outpaced that of causality, possibly due to the greater difficulty in performing causal inference.
- Diverse XAI Study Types: The research landscape includes a significant proportion of papers evaluating or applying existing XAI methods, followed by those introducing novel methods, discussions, reviews, and datasets/tools.
- Citation Trends: Review papers and novel methods tend to attract the most citations, indicating their perceived importance, while conceptual discussions also garner significant attention, reflecting ongoing debates in the field.
Key Results
- A cohort of 1,276 peer-reviewed papers on biomedical XAI was identified.
- An inflection point in publication rates was observed in October 2020, suggesting a significant acceleration in the field.
- Approximately 90% of the identified papers did not mention COVID-19, while 8% used COVID-19 data, and 3% mentioned COVID-19 as a driving force for their research.
- The study estimates that the field of biomedical XAI was pushed forward by about 25 months due to the pandemic.
- 50.4% of the papers evaluated or applied existing XAI methods, while 17% introduced novel XAI technologies.
- Reviews and novel methods attracted the highest average number of citations per month.
Limitations and Future Work
- Causality of the COVID-19 Link: While the authors hypothesize that COVID-19 accelerated XAI research, they acknowledge that establishing a definitive causal link would require a larger-scale analysis of multiple research topics.
- Defining Explainability: The paper notes the ongoing challenge of defining and quantifying "explainability," which remains a subject of debate.
- Future Directions: The authors suggest continued research into fostering clinical trust with interpretable machine learning models. They also point to the rise of large-scale foundation models (like GPT-3 and ChatGPT) and their potential to assist with clinical reasoning and uncover novel insights, suggesting a future where XAI might also involve understanding emergent properties from vast datasets. The debate on the trade-off between performance and interpretability, and the potential for explainable models to match or exceed the performance of black-box models, remains an active area.
Summary updated