Trust, Trustworthiness, and the Future of Medical AI: Outcomes of an Interdisciplinary Expert Workshop

Melanie Goisauf, Mónica Cano Abadía, Kaya Akyüz, Maciej Bobowicz, Alena Buyx, Ilaria Anna Colussi, Marie-Christine Fritzsche, Karim Lekadir, Pekka Marttinen, Michaela Th. Mayrhofer, János Mészáros

TL;DR

This paper argues that building trustworthy AI in healthcare requires a shift from purely technical solutions to a human-centered, interdisciplinary approach. It emphasizes the crucial roles of knowledge, human involvement, and stakeholder engagement throughout the AI lifecycle, suggesting that trust is a dynamic, relational process, not just a technical feature.

What Problem Does This Paper Address?

The paper tackles the challenge of ensuring that Artificial Intelligence (AI) in medicine is not only technically sound but also ethically robust and socially responsible. While existing guidelines for trustworthy AI often focus on technical principles like fairness and explainability, they frequently overlook broader human, institutional, and societal factors. This oversight can lead to AI systems that, despite their technical merits, fail to gain the trust of users or exacerbate existing inequalities. The authors aim to bridge this gap by proposing a more practice-oriented understanding of trust and trustworthiness in medical AI, grounded in real-world challenges.

Methods

The core of this paper's methodology is an interdisciplinary expert workshop. Fourteen academic experts from diverse fields (data science, law, medicine, ethics, social science, philosophy) participated in a two-day workshop focused on implementing AI ethics, particularly in oncological imaging and genomics. The workshop combined presentations and discussions covering ethical, legal, societal, technical, and clinical aspects of medical AI. Inspired by focus group and stakeholder engagement methods, the discussions were recorded and analyzed thematically. Additionally, participants were invited to complete an online questionnaire to gather further insights. The analysis team then interpreted these inputs to identify key themes and requirements for trustworthy AI.

Datasets

Not discussed in the paper. The paper focuses on conceptual and practical insights derived from expert discussions rather than empirical evaluation of AI models on specific datasets.

Key Insights and Ideas

  • Trust is Relational, Not Just Technical: Trustworthiness in medical AI isn't solely about algorithmic performance; it's a dynamic process involving humans, their expertise, and the social/institutional contexts.
  • Human Involvement is Paramount: Beyond technical solutions, knowledge and human involvement are essential. This includes understanding the AI's limitations, biases, and the specific context of its use.
  • Stakeholder Engagement is Crucial: Continuous engagement of all stakeholders (developers, clinicians, patients, ethicists, etc.) throughout the AI lifecycle is vital for building trust and ensuring AI systems are human- and multicentered.
  • Meaningful Information Over Technical Detail: Users (especially clinicians and patients) need meaningful, user-oriented information and a balance between knowledge and acceptable uncertainty, rather than just deep technical explanations.
  • Legal Frameworks Can Foster Trust: While trust isn't inherently a legal concept, regulations like the EU AI Act can define standards and procedures that promote trustworthiness and accountability.
  • Context Matters: Trust in AI is situated and dynamic, shaped by interactions and influenced by past experiences and societal expectations.

Key Results

The workshop and subsequent analysis revealed several key findings:

  • Beyond Bias and Fairness: While bias and fairness are critical, they are not the sole determinants of trustworthy AI. Knowledge and human involvement are equally, if not more, important.
  • User-Oriented Information: Experts emphasized the need for meaningful, context-specific information that helps users understand the AI's functionality, limitations, and uncertainty.
  • Confidence in Functionality: A core aspect of trust is the confidence that the AI tool will perform as expected.
  • Human-Centered Framework: The development of trustworthy AI should be guided by a human-centered, multistakeholder approach, ensuring diverse and inclusive engagement.
  • Recommendations for AI Lifecycle: The paper outlines specific recommendations for involving stakeholders at each stage of the AI lifecycle: design, development, validation, and deployment.

Limitations and Future Work

The authors acknowledge that while the EU AI Act provides a legal framework for trustworthiness, its interpretation and implementation across different member states, especially in clinical contexts, requires further examination. They also highlight the need for empirical studies to assess whether regulatory compliance translates into perceived trust among clinicians and patients. Future research should focus on developing new methodologies for measuring trust that go beyond quantitative metrics and incorporate thorough qualitative assessments to allow for agile management of trust-building processes. Additionally, the paper notes that while foundation models are a growing area, the aspects of trust discussed are relevant to their development and deployment.

Summary updated

Abstract

Trustworthiness has become a key concept for the ethical development and application of artificial intelligence (AI) in medicine. Various guidelines have formulated key principles, such as fairness, robustness, and explainability, as essential components to achieve trustworthy AI. However, conceptualizations of trustworthy AI often emphasize technical requirements and computational solutions, frequently overlooking broader aspects of fairness and potential biases. These include not only algorithmic bias but also human, institutional, social, and societal factors, which are critical to foster AI systems that are both ethically sound and socially responsible. This viewpoint article presents an interdisciplinary approach to analyzing trust in AI and trustworthy AI within the medical context, focusing on (1) social sciences and humanities conceptualizations and legal perspectives on trust and (2) their implications for trustworthy AI in health care. It focuses on real-world challenges in medicine that are often underrepresented in theoretical discussions to propose a more practice-oriented understanding. Insights were gathered from an interdisciplinary workshop with experts from various disciplines involved in the development and application of medical AI, particularly in oncological imaging and genomics, complemented by theoretical approaches related to trust in AI. Results emphasize that, beyond common issues of bias and fairness, knowledge and human involvement are essential for trustworthy AI. Stakeholder engagement throughout the AI life cycle emerged as crucial, supporting a human- and multicentered framework for trustworthy AI implementation. Findings emphasize that trust in medical AI depends on providing meaningful, user-oriented information and balancing knowledge with acceptable uncertainty. Experts highlighted the importance of confidence in the tool's functionality, specifically that it performs as expected. Trustworthiness was shown to be not a feature but rather a relational process, involving humans, their expertise, and the broader social or institutional contexts in which AI tools operate. Trust is dynamic, shaped by interactions among individuals, technologies, and institutions, and ultimately centers on people rather than tools alone. Tools are evaluated based on reliability and credibility, yet trust fundamentally relies on human connections. The article underscores the development of AI tools that are not only technically sound but also ethically robust and broadly accepted by end users, contributing to more effective and equitable AI-mediated health care. Findings highlight that building AI trustworthiness in health care requires a human-centered, multistakeholder approach with diverse and inclusive engagement. To promote equity, we recommend that AI development teams involve all relevant stakeholders at every stage of the AI lifecycle-from conception, technical development, clinical validation, and real-world deployment.
BibTeX
@article{goisauf2025trust,
  author = {Melanie Goisauf and Mónica Cano Abadía and Kaya Akyüz and Maciej Bobowicz and Alena Buyx and Ilaria Anna Colussi and Marie-Christine Fritzsche and Karim Lekadir and Pekka Marttinen and Michaela Th. Mayrhofer and János Mészáros},
  title = {{Trust, Trustworthiness, and the Future of Medical AI: Outcomes of an Interdisciplinary Expert Workshop}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/71236},
}
Figure