Abstract
The volume of manuscripts presenting machine learning (ML) models in medical literature, particularly within JMIR Publications journals, has seen a dramatic increase. However, inconsistent reporting standards often lead to prolonged peer-review cycles and challenges in reproducibility. To address this, the editors of JMIR Publications have introduced the Consolidated Reporting of Machine Learning Studies (CREMLS) guidelines. This unified checklist is designed to ensure transparency and rigor in the reporting of diagnostic and prognostic ML models, facilitating a more efficient review process and higher scientific quality.
The Reporting Crisis in Medical AI
The rapid integration of artificial intelligence into clinical research has created a "Wild West" of reporting. Since 2015, AI-related submissions to JMIR journals have increased 14-fold. Despite this surge, many manuscripts lack critical methodological details, such as how hyperparameters were tuned, how data leakage was mitigated, or how "data shift" (temporal changes in data) was addressed.
This lack of transparency results in "reviewer ping-pong," where editors and authors undergo multiple exhausting cycles simply to establish basic facts about the study's design. Without standardized reporting, it is nearly impossible for the scientific community to reproduce results or for clinicians to evaluate the safety and efficacy of models for real-world deployment.
Methodology: A Meta-Guideline Approach
CREMLS is not a new set of arbitrary rules but a "meta-guideline" developed through a structured literature review and expert curation. The authors identified existing reporting checklists and consolidated them into a single, comprehensive framework specifically for prognostic and diagnostic ML.
To ensure the guidelines are actionable, the paper "operationalizes" the checklist by providing concrete examples from high-quality published studies. This approach shows authors exactly what transparent reporting looks like in practice, moving beyond theoretical requirements to practical application.
The Five Domains of the CREMLS Checklist
The CREMLS framework organizes reporting requirements into five critical domains:
- Study Details: Clearly defining the clinical task, the intended use case, and necessary ethical approvals.
- The Data: Detailing inclusion and exclusion criteria, potential biases in data collection, and all preprocessing steps (e.g., imputation of missing values).
- Methodology: Providing a clear rationale for algorithm selection, strategies for handling class imbalance (such as SMOTE), and the specifics of hyperparameter optimization.
- Evaluation: Moving beyond simple accuracy to report performance metrics that consider the "cost of errors" and the model's sensitivity to data shift.
- Explainability: Ensuring that model outputs are plausible and interpretable by end-users, often through techniques like SHAP (SHapley Additive exPlanations) values.
Strategic Impact and Future Directions
JMIR Publications is officially adopting these guidelines, signaling a shift where standardized reporting is becoming a prerequisite for publication rather than a suggestion. By following this framework, researchers can directly accelerate the peer-review cycle and enhance the credibility of their findings.
Currently, CREMLS focuses on structured data (such as electronic health record tables) evaluated in "in silico" or simulated environments. The authors acknowledge that future iterations will need to expand into medical imaging (computer vision), clinical notes (natural language processing), and the complexities of live clinical deployment. As the field matures, the editors plan to refine the checklist based on implementation feedback to balance the burden of reporting with the necessity of transparency.
Summary updated