Piloting a Survey-Based Assessment of Transparency and Trustworthiness with Three Medical AI Tools

Jana Fehr, Giovanna Jaramillo-Gutierrez, Luis Oala, Matthias I. Gröschel, Manuel Bierwirth, Pradeep Balachandran, Alixandro Werneck-Leite, Christoph LippertHealthcare 2022 · DOI

TL;DR

This paper introduces and pilots a novel survey-based framework to quantify the transparency and trustworthiness of medical AI tools. The study found significant reporting gaps across three commercial AI use cases, with transparency scores ranging from 59% to 67%, directly impacting their trustworthiness and highlighting challenges posed by business constraints.

What Problem Does This Paper Address?

The increasing integration of Artificial Intelligence (AI) into healthcare promises significant advancements, but also introduces risks if AI models are poorly trained or validated. Ethical guidelines, such as those from the WHO and the EU's AI-HLEG, emphasize transparency in model development and validation as a cornerstone of trustworthy AI. However, a persistent issue is the common lack of transparent reporting for medical AI tools, making it difficult for stakeholders (clinicians, regulators, patients) to assess their reliability and safety. Existing guidance provides a roadmap for transparency, but there's a need for a practical method to quantify this transparency and its direct link to trustworthiness, especially for external assessments. This paper aims to bridge this gap by developing and piloting a framework to measure the transparency of medical AI tools.

Methods

The authors developed a comprehensive survey designed to elicit detailed information about medical AI use cases. This survey covers key areas such as:

  • Intended Use: What the AI is designed to do and its role (assistive vs. autonomous).
  • ML Technology: Details about the algorithms used, architecture, and training methodology.
  • Training Data: Information on data sources, collection, preprocessing, and ethical considerations.
  • Ethical Considerations: Data anonymization, consent, fairness, and potential harm.
  • Technical Validation: Performance metrics, external validation, and quality assessment.
  • Caveats and Recommendations: Limitations and advice for deployment.

The survey consists of semi-open questions, allowing for both structured and free-text responses. After initial survey completion, a teleconference was held with respondents to clarify answers and address any "not able to disclose" responses.

The core of their novel approach lies in the scoring mechanism:

  • Transparency Scoring: Each response was scored on a three-point scale (0, 0.5, or 1) indicating if the information was not, partially, or fully provided.
  • Trustworthiness Scoring: An analogous three-point scale was used to assess if the provided responses met the transparency requirements for trustworthiness criteria derived from ethical guidelines.

These scores were then aggregated to calculate overall transparency and trustworthiness percentages for each use case and across different sections of the survey.

Datasets

The study did not introduce new datasets. Instead, it focused on evaluating the transparency of reporting for existing medical AI use cases. The three use cases analyzed utilized:

  • UC1: A prediction model for detecting cancer in histopathology images, developed at a large company in Germany.
  • UC2: A multiclass model for predicting the correct placement of endotracheal or gastric tubes using X-rays, developed at a small company in India.
  • UC3: A model for routine care screening that predicts one type of cancer on X-rays, commercially available and developed at a small company in Germany.

The training data for these use cases involved a mix of publicly available datasets, purchased data, and preprocessed open-source datasets.

Key Insights and Ideas

  • Transparency is a direct precursor to Trustworthiness: The study clearly demonstrates that a lack of transparency in reporting directly leads to lower trustworthiness scores. If you can't see how it was built or validated, it's hard to trust it.
  • Business Constraints are Major Roadblocks: A recurring theme was that "business constraints" (e.g., protecting intellectual property, competitive advantage) significantly hindered the ability of companies to provide complete transparency, even when they might have had the information.
  • External Audits Face Feasibility Challenges: The findings suggest that conducting external audits for transparency, especially in the pre-market phase, is difficult if companies are unwilling or unable to disclose proprietary information. This raises questions about how to balance business secrets with the need for public trust.
  • Existing Guidelines are Not Universally Applied: Despite numerous guidelines for transparent reporting, the pilot study revealed that even commercial AI tools struggle to meet these standards.

Key Results

The pilot assessment of three medical AI use cases yielded the following results:

  • Overall Transparency Scores:
    • UC2: 67.2%
    • UC3: 67.2%
    • UC1: 59.0%
  • Overall Trustworthiness Scores:
    • UC3: 64.3%
    • UC2: 52.4%
    • UC1: 48.8%
  • Section-Specific Performance: Scores varied significantly by section. For instance, "Intended Use" generally scored higher than "Technical Validation and Quality Assessment" or "Caveats and Recommendations for Deployment."
  • Qualitative Findings: Anecdotal evidence pointed to issues like limited information from external datasets, difficulties in reporting ensemble models, and challenges in clarifying deployment caveats.

Limitations and Future Work

The authors acknowledge several limitations:

  • Small Sample Size: The study only involved three use cases, limiting the generalizability of the findings to the broader medical AI landscape.
  • Subjectivity of Scoring: The scoring was performed by a multidisciplinary team, and while efforts were made for consistency, some degree of subjectivity is inherent. A larger team or different scoring scales (e.g., 5- or 10-point) could be explored.
  • Verification of Truthfulness: The study relied on self-reported information, and there was no independent verification of the truthfulness of the responses.
  • Survey Adaptability: The survey might need adjustments for different types of AI algorithms (beyond supervised ML) or specific stakeholder requirements.

Future work suggested includes:

  • Refining the Survey: Incorporating feedback from a larger group of stakeholders to improve its comprehensiveness and adaptability.
  • Larger-Scale Administration: Applying the survey to a broader sample of companies to confirm the observed trends and challenges.
  • Developing New Strategies: Investigating novel approaches to enable external audits while respecting business secrets.
  • Policy Development: Advocating for policies that define minimum transparency requirements for trustworthy medical AI.

Summary updated

Abstract

Artificial intelligence (AI) offers the potential to support healthcare delivery, but poorly trained or validated algorithms bear risks of harm. Ethical guidelines stated transparency about model development and validation as a requirement for trustworthy AI. Abundant guidance exists to provide transparency through reporting, but poorly reported medical AI tools are common. To close this transparency gap, we developed and piloted a framework to quantify the transparency of medical AI tools with three use cases. Our framework comprises a survey to report on the intended use, training and validation data and processes, ethical considerations, and deployment recommendations. The transparency of each response was scored with either 0, 0.5, or 1 to reflect if the requested information was not, partially, or fully provided. Additionally, we assessed on an analogous three-point scale if the provided responses fulfilled the transparency requirement for a set of trustworthiness criteria from ethical guidelines. The degree of transparency and trustworthiness was calculated on a scale from 0% to 100%. Our assessment of three medical AI use cases pin-pointed reporting gaps and resulted in transparency scores of 67% for two use cases and one with 59%. We report anecdotal evidence that business constraints and limited information from external datasets were major obstacles to providing transparency for the three use cases. The observed transparency gaps also lowered the degree of trustworthiness, indicating compliance gaps with ethical guidelines. All three pilot use cases faced challenges to provide transparency about medical AI tools, but more studies are needed to investigate those in the wider medical AI sector. Applying this framework for an external assessment of transparency may be infeasible if business constraints prevent the disclosure of information. New strategies may be necessary to enable audits of medical AI tools while preserving business secrets.
BibTeX
@inproceedings{fehr2022piloting,
  author = {Jana Fehr and Giovanna Jaramillo-Gutierrez and Luis Oala and Matthias I. Gröschel and Manuel Bierwirth and Pradeep Balachandran and Alixandro Werneck-Leite and Christoph Lippert},
  title = {{Piloting a Survey-Based Assessment of Transparency and Trustworthiness with Three Medical AI Tools}},
  booktitle = {Healthcare},
  year = {2022},
  doi = {10.3390/healthcare10101923},
}
Figure