Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View

Wei Luo, Dinh Phung, Truyen Tran, Sunil Gupta, Santu Rana, Chandan Karmakar, Alistair Shilton, John Yearwood, Nevenka Dimitrova, Tu Bao Ho, Svetha Venkatesh, Michael BerkJournal of Medical Internet Research 2016 · DOI

TL;DR

Machine learning is increasingly used in biomedical research, but its complexity leads to inconsistent reporting and potential misuse. This paper addresses this by proposing a multidisciplinary set of guidelines for developing and reporting machine learning predictive models, aiming to improve reproducibility and accelerate the adoption of these powerful tools in clinical settings.

What Problem Does This Paper Address?

The explosion of "big data" in healthcare and biomedical research presents immense opportunities for discovery. Machine learning (ML) models are at the forefront of analyzing this data, offering powerful predictive capabilities that surpass traditional statistical methods. However, the inherent complexity of ML algorithms, coupled with the flexibility in how they can be implemented and reported, often leads to a "black box" problem. This results in inconsistent application, insufficient reporting of crucial details, and a lack of transparency, making it difficult to assess model validity, distinguish genuine discoveries from chance findings, and ultimately hindering the reliable integration of ML into clinical practice.

Methods

The authors employed a multidisciplinary approach, specifically the Delphi method, to develop their guidelines. This involved assembling a panel of 11 experts from machine learning, traditional statistics, and biomedical applications. The process was iterative, starting with broad topic suggestions and progressively refining them through multiple rounds of email interviews. The panel aimed to create two key outputs: (1) a comprehensive list of reporting items for research articles and (2) a set of practical sequential steps for developing predictive models. The guidelines distinguish between "essential" items that should always be reported and "desirable" items that are recommended when applicable.

Datasets

Not discussed in the paper. The paper focuses on the methodology for developing and reporting ML models rather than evaluating specific models on particular datasets.

Key Insights and Ideas

  • The "Black Magic" of ML: The paper highlights that the complexity of ML methods can lead to them being perceived as "black magic," making them prone to misuse and difficult for non-experts to understand or validate.
  • The Need for Standardization: The core insight is the urgent need for standardized guidelines to ensure the correct application and consistent reporting of ML models in biomedical research. This is crucial for building trust and facilitating the translation of ML findings into clinical practice.
  • Multidisciplinary Collaboration is Key: The development of these guidelines emphasizes the importance of bringing together experts from diverse fields (ML, statistics, clinical practice) to address the multifaceted challenges of ML in biomedicine.
  • Distinguishing Essential from Desirable: The approach of categorizing reporting items into "essential" and "desirable" provides a practical framework for researchers, allowing for flexibility while ensuring critical information is not omitted.
  • Beyond Prediction Accuracy: The guidelines implicitly acknowledge that while prediction accuracy is important, interpretability, generalizability, and understanding the clinical implications are equally vital for the successful adoption of ML models.

Key Results

The primary result of this work is the generation of a comprehensive set of guidelines for developing and reporting machine learning predictive models in biomedical research. These guidelines are presented in several tables and figures within the paper, covering:

  • Reporting Items: Detailed checklists for the title, abstract, introduction, methods, results, and discussion sections of research articles. These items cover aspects like study design, prediction problem definition, data preparation, model building, validation strategies, performance metrics, and clinical implications.
  • Development Steps: A flowchart illustrating the information flow in the predictive modeling process, including data splitting, training, testing, and external validation.
  • Addressing Common Pitfalls: The guidelines also implicitly or explicitly address common issues like data leakage and overfitting through specific reporting requirements and development steps.

While quantitative results (e.g., specific accuracy improvements) are not the focus, the qualitative outcome is a structured framework designed to improve the quality and reproducibility of ML research in biomedicine.

Limitations and Future Work

The authors acknowledge that ML is a rapidly evolving field, and their guidelines are not exhaustive. They are expected to evolve as research progresses. Future work would likely involve the broader adoption and refinement of these guidelines by the research community, potentially leading to empirical studies validating their impact on the quality of published ML research in biomedicine. The paper also notes that the guidelines cover "most popular" ML methods, implying that less common or emerging techniques might require further consideration.

Summary updated

Abstract

A set of guidelines was generated to enable correct application of machine learning models and consistent reporting of model specifications and results in biomedical research. We believe that such guidelines will accelerate the adoption of big data analysis, particularly with machine learning methods, in the biomedical research community.
BibTeX
@article{luo2016guidelines,
  author = {Wei Luo and Dinh Phung and Truyen Tran and Sunil Gupta and Santu Rana and Chandan Karmakar and Alistair Shilton and John Yearwood and Nevenka Dimitrova and Tu Bao Ho and Svetha Venkatesh and Michael Berk},
  title = {{Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View}},
  journal = {Journal of Medical Internet Research},
  year = {2016},
  doi = {10.2196/jmir.5870},
}
Figure