Low adherence to existing model reporting guidelines by commonly used clinical prediction models

Jonathan Lu, Alison Callahan, Birju Patel, Keith Morse, Dev Dash, Nigam H. ShahmedRxiv 2021 · DOI

Abstract

Objective: To evaluate whether documentation for widely used machine learning models developed by a leading electronic health record (EHR) vendor meets the standards set by established model reporting guidelines.

Materials and Methods: Researchers identified 220 unique reporting items ("atoms") across 15 guidelines from computer science and biomedical informatics. They audited the documentation for 12 commonly deployed models from Epic Systems. Four independent reviewers and one adjudicator assessed the presence of these atoms in the vendor's "Model Briefs."

Results: The median completion rate for applicable reporting atoms was 39% (range: 31%–47%). While basic metrics like AUROC and outcome definitions were frequently reported (90%+), critical information regarding fairness, external validation, and model transparency was missing in over half of the reviewed documentation. Median adherence to any single entire guideline was only 54%.

Conclusion: Documentation for clinical predictive models is currently insufficient for users to fully assess model reliability, fairness, and clinical utility. There is a pressing need for better operationalization of reporting standards in healthcare AI.

Introduction: The "AI Chasm" in Healthcare

In recent years, machine learning models have been integrated into healthcare workflows to predict everything from sepsis to patient readmission. However, a significant "AI chasm" exists—a gap where models that perform well in research settings fail to produce clinical gains or equitable outcomes in practice.

A primary driver of this chasm is the lack of transparent documentation. If a clinical site does not know how a model handles missing data, whether it was validated on a diverse population, or what its underlying coefficients are, they cannot safely or effectively deploy it. While various academic bodies have published reporting guidelines (checklists of what developers should disclose), it has remained unclear to what extent commercial vendors adhere to these standards.

Methods

The researchers conducted a systematic audit of model documentation using a four-step process:

  1. Atomization: They analyzed 15 major reporting guidelines—including TRIPOD, Model Cards, and CONSORT-AI—and decomposed them into 220 unique "atoms," or individual pieces of requested information.
  2. Mapping: These atoms were categorized by model lifecycle stages, such as "Model Development," "Fairness," and "Prospective Evaluation."
  3. The Audit: The team reviewed the "Cognitive Computing Model Briefs" (the primary documentation provided to customers) for 12 of the most frequently used predictive models from Epic Systems, the dominant EHR vendor in the United States.
  4. Review and Adjudication: Four independent reviewers scored the presence of each atom. Discrepancies were resolved through a formal adjudication process to ensure a consensus-based final score.

Key Insights

The "Basics" Bias

The audit revealed that vendors are proficient at reporting "what" a model does but poor at explaining the "how" or "for whom." Documentation almost universally included the outcome definition and the Area Under the Receiver Operating Characteristic curve (AUROC). However, the actual model coefficients or weights—essential for transparency—were reported in only 8.3% of the briefs.

The Fairness and Reliability Gap

Despite the high stakes of algorithmic bias in medicine, information regarding fairness was missing from at least half of the documentation. Subgroup analyses by age, sex, or race/ethnicity were rarely performed or disclosed. Furthermore, reliability metrics were notably low: only 33% of models reported results from external validation, and 0% provided confidence intervals or statistical significance for their performance metrics.

Guideline Overload

With 220 unique items across 15 guidelines, the authors suggest that the sheer volume of recommendations may be "formidable" for developers. No single guideline currently covers the entire model lifecycle perfectly. For instance, while Model Cards excels at fairness reporting, TRIPOD provides better coverage of development details. The Model Facts Labels showed the highest adherence (74%), suggesting that simplified, standardized formats may be more effective than exhaustive checklists.

Results Summary

  • Median Completion Rate: 39% of applicable reporting atoms were present.
  • Guideline Adherence: The median adherence to individual guidelines was 54%.
  • Transparency Issues: Only 8.3% of models provided saliency maps or coefficients.
  • Clinical Utility: 0% of the documentation provided user-facing materials or specific directions for clinicians on how to interpret the model at the point of care.

Discussion and Future Work

The study's findings suggest that even with "generous credit"—such as counting a prompt to "contact a representative" as a reported strategy—commercial documentation falls short of professional standards. This lack of transparency prevents clinicians from identifying when a model might be out-of-scope or biased against specific patient demographics.

To bridge the AI chasm, the authors call for the creation of a "public dashboard" to track model adherence to reporting guidelines, similar to the tracking of clinical trials. Moving forward, professional societies must move beyond simply publishing guidelines and begin endorsing specific, operationalized reporting practices to incentivize vendors to move away from the "black box" approach.

Summary updated

Abstract

ABSTRACT Objective To assess whether the documentation available for commonly used machine learning models developed by an electronic health record (EHR) vendor provides information requested by model reporting guidelines. Materials and Methods We identified items requested for reporting from model reporting guidelines published in computer science, biomedical informatics, and clinical journals, and merged similar items into representative “atoms”. Four independent reviewers and one adjudicator assessed the degree to which model documentation for 12 models developed by Epic Systems reported the details requested in each atom. We present summary statistics of consensus, interrater agreement, and reporting rates of all atoms for the 12 models. Results We identified 220 unique atoms across 15 model reporting guidelines. After examining the documentation for the 12 most commonly used Epic models, the independent reviewers had an interrater agreement of 76%. After adjudication, the model documentations’ median completion rate of applicable atoms was 39% (range: 31%-47%). Most of the commonly requested atoms had reporting rates of 90% or above, including atoms concerning outcome definition, preprocessing, AUROC, internal validation and intended clinical use. For individual reporting guidelines, the median adherence rate for an entire guideline was 54% (range: 15%-71%). Atoms reported half the time or less included those relating to fairness (summary statistics and subgroup analyses, including for age, race/ethnicity, or sex), usefulness (net benefit, prediction time, warnings on out-of-scope use and when to stop use), and transparency (model coefficients). Atoms relating to reliability also had low reporting, including those related to missingness (missing data statistics, missingness strategy), validation (calibration plot, external validation), and monitoring (how models are updated/tuned, prediction monitoring). Conclusion There are many recommendations about what should be reported about predictive models used to guide care. Existing model documentation examined in this study provides less than half of applicable atoms, and entire reporting guidelines have low adherence rates. Half or less of the reviewed documentation reported information related to usefulness, reliability, transparency and fairness of models. There is a need for better operationalization of reporting recommendations for predictive models in healthcare. KEY POINTS Question How often does documentation for commonly deployed clinical predictive models report the information requested by model reporting guidelines? Finding Combining the recommendations from 15 model reporting guidelines, we identified 220 unique requested items. We reviewed the documentation of 12 commonly deployed Epic models and assessed the completion rate of applicable items. The median completion rate was 39%. While the most commonly requested items were highly reported, information on usefulness, reliability, transparency and fairness was missing from at least half of documentation. Meaning There is incomplete documentation for model users to ensure that deployed models are useful, reliable, transparent and fair.
BibTeX
@misc{lu2021low,
  author = {Jonathan Lu and Alison Callahan and Birju Patel and Keith Morse and Dev Dash and Nigam H. Shah},
  title = {{Low adherence to existing model reporting guidelines by commonly used clinical prediction models}},
  year = {2021},
  doi = {10.1101/2021.07.21.21260282},
  howpublished = {medRxiv},
}
Figure