TL;DR
This paper introduces and pilots a novel survey-based framework to quantify the transparency and trustworthiness of medical AI tools. The study found significant reporting gaps across three commercial AI use cases, with transparency scores ranging from 59% to 67%, directly impacting their trustworthiness and highlighting challenges posed by business constraints.
What Problem Does This Paper Address?
The increasing integration of Artificial Intelligence (AI) into healthcare promises significant advancements, but also introduces risks if AI models are poorly trained or validated. Ethical guidelines, such as those from the WHO and the EU's AI-HLEG, emphasize transparency in model development and validation as a cornerstone of trustworthy AI. However, a persistent issue is the common lack of transparent reporting for medical AI tools, making it difficult for stakeholders (clinicians, regulators, patients) to assess their reliability and safety. Existing guidance provides a roadmap for transparency, but there's a need for a practical method to quantify this transparency and its direct link to trustworthiness, especially for external assessments. This paper aims to bridge this gap by developing and piloting a framework to measure the transparency of medical AI tools.
Methods
The authors developed a comprehensive survey designed to elicit detailed information about medical AI use cases. This survey covers key areas such as:
- Intended Use: What the AI is designed to do and its role (assistive vs. autonomous).
- ML Technology: Details about the algorithms used, architecture, and training methodology.
- Training Data: Information on data sources, collection, preprocessing, and ethical considerations.
- Ethical Considerations: Data anonymization, consent, fairness, and potential harm.
- Technical Validation: Performance metrics, external validation, and quality assessment.
- Caveats and Recommendations: Limitations and advice for deployment.
The survey consists of semi-open questions, allowing for both structured and free-text responses. After initial survey completion, a teleconference was held with respondents to clarify answers and address any "not able to disclose" responses.
The core of their novel approach lies in the scoring mechanism:
- Transparency Scoring: Each response was scored on a three-point scale (0, 0.5, or 1) indicating if the information was not, partially, or fully provided.
- Trustworthiness Scoring: An analogous three-point scale was used to assess if the provided responses met the transparency requirements for trustworthiness criteria derived from ethical guidelines.
These scores were then aggregated to calculate overall transparency and trustworthiness percentages for each use case and across different sections of the survey.
Datasets
The study did not introduce new datasets. Instead, it focused on evaluating the transparency of reporting for existing medical AI use cases. The three use cases analyzed utilized:
- UC1: A prediction model for detecting cancer in histopathology images, developed at a large company in Germany.
- UC2: A multiclass model for predicting the correct placement of endotracheal or gastric tubes using X-rays, developed at a small company in India.
- UC3: A model for routine care screening that predicts one type of cancer on X-rays, commercially available and developed at a small company in Germany.
The training data for these use cases involved a mix of publicly available datasets, purchased data, and preprocessed open-source datasets.
Key Insights and Ideas
- Transparency is a direct precursor to Trustworthiness: The study clearly demonstrates that a lack of transparency in reporting directly leads to lower trustworthiness scores. If you can't see how it was built or validated, it's hard to trust it.
- Business Constraints are Major Roadblocks: A recurring theme was that "business constraints" (e.g., protecting intellectual property, competitive advantage) significantly hindered the ability of companies to provide complete transparency, even when they might have had the information.
- External Audits Face Feasibility Challenges: The findings suggest that conducting external audits for transparency, especially in the pre-market phase, is difficult if companies are unwilling or unable to disclose proprietary information. This raises questions about how to balance business secrets with the need for public trust.
- Existing Guidelines are Not Universally Applied: Despite numerous guidelines for transparent reporting, the pilot study revealed that even commercial AI tools struggle to meet these standards.
Key Results
The pilot assessment of three medical AI use cases yielded the following results:
- Overall Transparency Scores:
- UC2: 67.2%
- UC3: 67.2%
- UC1: 59.0%
- Overall Trustworthiness Scores:
- UC3: 64.3%
- UC2: 52.4%
- UC1: 48.8%
- Section-Specific Performance: Scores varied significantly by section. For instance, "Intended Use" generally scored higher than "Technical Validation and Quality Assessment" or "Caveats and Recommendations for Deployment."
- Qualitative Findings: Anecdotal evidence pointed to issues like limited information from external datasets, difficulties in reporting ensemble models, and challenges in clarifying deployment caveats.
Limitations and Future Work
The authors acknowledge several limitations:
- Small Sample Size: The study only involved three use cases, limiting the generalizability of the findings to the broader medical AI landscape.
- Subjectivity of Scoring: The scoring was performed by a multidisciplinary team, and while efforts were made for consistency, some degree of subjectivity is inherent. A larger team or different scoring scales (e.g., 5- or 10-point) could be explored.
- Verification of Truthfulness: The study relied on self-reported information, and there was no independent verification of the truthfulness of the responses.
- Survey Adaptability: The survey might need adjustments for different types of AI algorithms (beyond supervised ML) or specific stakeholder requirements.
Future work suggested includes:
- Refining the Survey: Incorporating feedback from a larger group of stakeholders to improve its comprehensiveness and adaptability.
- Larger-Scale Administration: Applying the survey to a broader sample of companies to confirm the observed trends and challenges.
- Developing New Strategies: Investigating novel approaches to enable external audits while respecting business secrets.
- Policy Development: Advocating for policies that define minimum transparency requirements for trustworthy medical AI.
Summary updated