TL;DR
This paper argues that building trustworthy AI in healthcare requires a shift from purely technical solutions to a human-centered, interdisciplinary approach. It emphasizes the crucial roles of knowledge, human involvement, and stakeholder engagement throughout the AI lifecycle, suggesting that trust is a dynamic, relational process, not just a technical feature.
What Problem Does This Paper Address?
The paper tackles the challenge of ensuring that Artificial Intelligence (AI) in medicine is not only technically sound but also ethically robust and socially responsible. While existing guidelines for trustworthy AI often focus on technical principles like fairness and explainability, they frequently overlook broader human, institutional, and societal factors. This oversight can lead to AI systems that, despite their technical merits, fail to gain the trust of users or exacerbate existing inequalities. The authors aim to bridge this gap by proposing a more practice-oriented understanding of trust and trustworthiness in medical AI, grounded in real-world challenges.
Methods
The core of this paper's methodology is an interdisciplinary expert workshop. Fourteen academic experts from diverse fields (data science, law, medicine, ethics, social science, philosophy) participated in a two-day workshop focused on implementing AI ethics, particularly in oncological imaging and genomics. The workshop combined presentations and discussions covering ethical, legal, societal, technical, and clinical aspects of medical AI. Inspired by focus group and stakeholder engagement methods, the discussions were recorded and analyzed thematically. Additionally, participants were invited to complete an online questionnaire to gather further insights. The analysis team then interpreted these inputs to identify key themes and requirements for trustworthy AI.
Datasets
Not discussed in the paper. The paper focuses on conceptual and practical insights derived from expert discussions rather than empirical evaluation of AI models on specific datasets.
Key Insights and Ideas
- Trust is Relational, Not Just Technical: Trustworthiness in medical AI isn't solely about algorithmic performance; it's a dynamic process involving humans, their expertise, and the social/institutional contexts.
- Human Involvement is Paramount: Beyond technical solutions, knowledge and human involvement are essential. This includes understanding the AI's limitations, biases, and the specific context of its use.
- Stakeholder Engagement is Crucial: Continuous engagement of all stakeholders (developers, clinicians, patients, ethicists, etc.) throughout the AI lifecycle is vital for building trust and ensuring AI systems are human- and multicentered.
- Meaningful Information Over Technical Detail: Users (especially clinicians and patients) need meaningful, user-oriented information and a balance between knowledge and acceptable uncertainty, rather than just deep technical explanations.
- Legal Frameworks Can Foster Trust: While trust isn't inherently a legal concept, regulations like the EU AI Act can define standards and procedures that promote trustworthiness and accountability.
- Context Matters: Trust in AI is situated and dynamic, shaped by interactions and influenced by past experiences and societal expectations.
Key Results
The workshop and subsequent analysis revealed several key findings:
- Beyond Bias and Fairness: While bias and fairness are critical, they are not the sole determinants of trustworthy AI. Knowledge and human involvement are equally, if not more, important.
- User-Oriented Information: Experts emphasized the need for meaningful, context-specific information that helps users understand the AI's functionality, limitations, and uncertainty.
- Confidence in Functionality: A core aspect of trust is the confidence that the AI tool will perform as expected.
- Human-Centered Framework: The development of trustworthy AI should be guided by a human-centered, multistakeholder approach, ensuring diverse and inclusive engagement.
- Recommendations for AI Lifecycle: The paper outlines specific recommendations for involving stakeholders at each stage of the AI lifecycle: design, development, validation, and deployment.
Limitations and Future Work
The authors acknowledge that while the EU AI Act provides a legal framework for trustworthiness, its interpretation and implementation across different member states, especially in clinical contexts, requires further examination. They also highlight the need for empirical studies to assess whether regulatory compliance translates into perceived trust among clinicians and patients. Future research should focus on developing new methodologies for measuring trust that go beyond quantitative metrics and incorporate thorough qualitative assessments to allow for agile management of trust-building processes. Additionally, the paper notes that while foundation models are a growing area, the aspects of trust discussed are relevant to their development and deployment.
Summary updated