Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness

Sebastian J. Vollmer, Bilal A. Mateen, Gergo Bohner, Franz J. Király, Rayid Ghani, Pall Jonsson, Sarah Cumbers, Adrian Jonas, Katherine S. L. McAllister, Puja Myles, David Granger, Mark Birse, Richard Branson, Karel G. M. Moons, Gary S. Collins, John P. A. Ioannidis, Chris C. Holmes, Harry HemingwayarXiv 2018 · OpenReview

Introduction

The integration of machine learning (ML) and artificial intelligence (AI) into healthcare promises a revolution in patient care, from automated diagnostics to personalized treatment plans. However, despite a surge in biomedical research demonstrating high accuracy in controlled environments, a significant gap remains between the "computer bench" and the clinical bedside. Many algorithms fail to transition into practice, and those that do often lack the rigorous scrutiny applied to traditional medical interventions like pharmaceuticals.

The primary obstacle is a lack of standardized, best-practice guidance tailored specifically to ML/AI in health. Without a robust evaluative framework, healthcare decision-makers struggle to distinguish between truly effective tools and those that are overhyped, biased, or potentially harmful. To address this, the AI-TREE framework was developed, focusing on four pillars: Transparency, Replicability, Ethics, and Effectiveness.

The AI-TREE Framework

Developed through collaboration between leading UK and international organizations—including the Alan Turing Institute, NICE, and the MHRA—the AI-TREE framework provides a qualitative evaluation structure. It is centered around the "health data-science cycle," guiding stakeholders through 20 critical questions across the project lifecycle.

The development cycle of ML/AI solutions for clinical care, highlighting the TREE principles.

The framework categorizes these questions into four distinct phases:

  1. Inception: Focuses on defining tangible patient benefits, ensuring meaningful involvement of patients and the public, and establishing transparency in data flow.
  2. Analysis: Evaluates data suitability, acknowledges real-world constraints, assesses computational resource needs, and ensures the modeling pipeline is reproducible with appropriate baselines.
  3. Impact Evaluation: Prioritizes testing for algorithmic bias and inequity, ensuring model interpretability, and providing proof of real-world effectiveness.
  4. Implementation: Addresses post-deployment monitoring, cost-effectiveness, the distribution of financial benefits, and regulatory compliance (such as GDPR and CE marking).

Critical Insights for Health AI Development

The framework highlights several systemic issues currently plaguing the field:

Clinical Utility Over Statistical Accuracy

A recurring theme is that "just because we can, doesn't mean we should." A model with high statistical accuracy is clinically useless if it does not inform a specific decision-making process. Developers must understand the clinical actions that will follow a model's output. For instance, the authors note that IBM’s Watson for Oncology was often evaluated on "concordance" (how often it agreed with a doctor) rather than actual improved patient outcomes.

The Problem of Weak Comparators

AI models are frequently compared against weak baselines or uninformed guesses rather than the current clinical gold standard. This "weak comparator" bias artificially inflates the perceived value of new tools. To be truly effective, AI must demonstrate superiority or non-inferiority against established, high-performing biostatistical models.

Dynamic Data and Model Degradation

Unlike static software, health AI models operate in dynamic environments. Changes in population demographics, hospital procedures, or the introduction of new treatments can cause model performance to degrade over time. Historical examples, such as early mammography computer-aided detection (CAD) systems, showed initial promise but failed to improve diagnostic accuracy in long-term national registries. Continuous post-deployment monitoring is essential.

Proactive Fairness and Ethics

Algorithmic bias is a significant risk. If training data underrepresents minority groups, the resulting AI can exacerbate existing health inequities. The AI-TREE framework mandates systematic testing for bias across protected characteristics like age, sex, and ethnicity before any deployment occurs.

Conclusion and Future Directions

Implementing a global framework for health AI faces challenges, including the need for common terminology across computer science and medicine, and balancing the need for transparency with intellectual property rights. However, the AI-TREE framework serves as a foundational step toward ensuring that AI innovation translates into genuine patient benefit. Future work involves a systematic review of the 20 questions to build an internationally agreed-upon consensus for AI in healthcare.

Summary updated

Abstract

Machine learning (ML), artificial intelligence (AI) and other modern statistical methods are providing new opportunities to operationalize previously untapped and rapidly growing sources of data for patient benefit. Whilst there is a lot of promising research currently being undertaken, the literature as a whole lacks: transparency; clear reporting to facilitate replicability; exploration for potential ethical concerns; and, clear demonstrations of effectiveness. There are many reasons for why these issues exist, but one of the most important that we provide a preliminary solution for here is the current lack of ML/AI- specific best practice guidance. Although there is no consensus on what best practice looks in this field, we believe that interdisciplinary groups pursuing research and impact projects in the ML/AI for health domain would benefit from answering a series of questions based on the important issues that exist when undertaking work of this nature. Here we present 20 questions that span the entire project life cycle, from inception, data analysis, and model evaluation, to implementation, as a means to facilitate project planning and post-hoc (structured) independent evaluation. By beginning to answer these questions in different settings, we can start to understand what constitutes a good answer, and we expect that the resulting discussion will be central to developing an international consensus framework for transparent, replicable, ethical and effective research in artificial intelligence (AI-TREE) for health.
BibTeX
@misc{vollmer2018machine,
  author = {Sebastian J. Vollmer and Bilal A. Mateen and Gergo Bohner and Franz J. Király and Rayid Ghani and Pall Jonsson and Sarah Cumbers and Adrian Jonas and Katherine S. L. McAllister and Puja Myles and David Granger and Mark Birse and Richard Branson and Karel G. M. Moons and Gary S. Collins and John P. A. Ioannidis and Chris C. Holmes and Harry Hemingway},
  title = {{Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness}},
  year = {2018},
  eprint = {1812.10404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.10404},
}
Figure