Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Models (CREMLS)

Khaled El Emam, Tiffany I. Leung, Bradley Malin, William Klement, Günther EysenbachJournal of Medical Internet Research 2024 · DOI · OpenAlex

Abstract

The volume of manuscripts presenting machine learning (ML) models in medical literature, particularly within JMIR Publications journals, has seen a dramatic increase. However, inconsistent reporting standards often lead to prolonged peer-review cycles and challenges in reproducibility. To address this, the editors of JMIR Publications have introduced the Consolidated Reporting of Machine Learning Studies (CREMLS) guidelines. This unified checklist is designed to ensure transparency and rigor in the reporting of diagnostic and prognostic ML models, facilitating a more efficient review process and higher scientific quality.

The Reporting Crisis in Medical AI

The rapid integration of artificial intelligence into clinical research has created a "Wild West" of reporting. Since 2015, AI-related submissions to JMIR journals have increased 14-fold. Despite this surge, many manuscripts lack critical methodological details, such as how hyperparameters were tuned, how data leakage was mitigated, or how "data shift" (temporal changes in data) was addressed.

This lack of transparency results in "reviewer ping-pong," where editors and authors undergo multiple exhausting cycles simply to establish basic facts about the study's design. Without standardized reporting, it is nearly impossible for the scientific community to reproduce results or for clinicians to evaluate the safety and efficacy of models for real-world deployment.

Methodology: A Meta-Guideline Approach

CREMLS is not a new set of arbitrary rules but a "meta-guideline" developed through a structured literature review and expert curation. The authors identified existing reporting checklists and consolidated them into a single, comprehensive framework specifically for prognostic and diagnostic ML.

To ensure the guidelines are actionable, the paper "operationalizes" the checklist by providing concrete examples from high-quality published studies. This approach shows authors exactly what transparent reporting looks like in practice, moving beyond theoretical requirements to practical application.

The Five Domains of the CREMLS Checklist

The CREMLS framework organizes reporting requirements into five critical domains:

  1. Study Details: Clearly defining the clinical task, the intended use case, and necessary ethical approvals.
  2. The Data: Detailing inclusion and exclusion criteria, potential biases in data collection, and all preprocessing steps (e.g., imputation of missing values).
  3. Methodology: Providing a clear rationale for algorithm selection, strategies for handling class imbalance (such as SMOTE), and the specifics of hyperparameter optimization.
  4. Evaluation: Moving beyond simple accuracy to report performance metrics that consider the "cost of errors" and the model's sensitivity to data shift.
  5. Explainability: Ensuring that model outputs are plausible and interpretable by end-users, often through techniques like SHAP (SHapley Additive exPlanations) values.

Strategic Impact and Future Directions

JMIR Publications is officially adopting these guidelines, signaling a shift where standardized reporting is becoming a prerequisite for publication rather than a suggestion. By following this framework, researchers can directly accelerate the peer-review cycle and enhance the credibility of their findings.

Currently, CREMLS focuses on structured data (such as electronic health record tables) evaluated in "in silico" or simulated environments. The authors acknowledge that future iterations will need to expand into medical imaging (computer vision), clinical notes (natural language processing), and the complexities of live clinical deployment. As the field matures, the editors plan to refine the checklist based on implementation feedback to balance the burden of reporting with the necessity of transparency.

Summary updated

Abstract

The number of papers presenting machine learning (ML) models that are being submitted to and published in the Journal of Medical Internet Research and other JMIR Publications journals has steadily increased. Editors and peer reviewers involved in the review process for such manuscripts often go through multiple review cycles to enhance the quality and completeness of reporting. The use of reporting guidelines or checklists can help ensure consistency in the quality of submitted (and published) scientific manuscripts and, for example, avoid instances of missing information. In this Editorial, the editors of JMIR Publications journals discuss the general JMIR Publications policy regarding authors' application of reporting guidelines and specifically focus on the reporting of ML studies in JMIR Publications journals, using the Consolidated Reporting of Machine Learning Studies (CREMLS) guidelines, with an example of how authors and other journals could use the CREMLS checklist to ensure transparency and rigor in reporting.
BibTeX
@article{emam2024consolidated,
  author = {Khaled El Emam and Tiffany I. Leung and Bradley Malin and William Klement and Günther Eysenbach},
  title = {{Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Models (CREMLS)}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/52508},
}
Figure