Abstract
Objective: To evaluate whether documentation for widely used machine learning models developed by a leading electronic health record (EHR) vendor meets the standards set by established model reporting guidelines.
Materials and Methods: Researchers identified 220 unique reporting items ("atoms") across 15 guidelines from computer science and biomedical informatics. They audited the documentation for 12 commonly deployed models from Epic Systems. Four independent reviewers and one adjudicator assessed the presence of these atoms in the vendor's "Model Briefs."
Results: The median completion rate for applicable reporting atoms was 39% (range: 31%–47%). While basic metrics like AUROC and outcome definitions were frequently reported (90%+), critical information regarding fairness, external validation, and model transparency was missing in over half of the reviewed documentation. Median adherence to any single entire guideline was only 54%.
Conclusion: Documentation for clinical predictive models is currently insufficient for users to fully assess model reliability, fairness, and clinical utility. There is a pressing need for better operationalization of reporting standards in healthcare AI.
Introduction: The "AI Chasm" in Healthcare
In recent years, machine learning models have been integrated into healthcare workflows to predict everything from sepsis to patient readmission. However, a significant "AI chasm" exists—a gap where models that perform well in research settings fail to produce clinical gains or equitable outcomes in practice.
A primary driver of this chasm is the lack of transparent documentation. If a clinical site does not know how a model handles missing data, whether it was validated on a diverse population, or what its underlying coefficients are, they cannot safely or effectively deploy it. While various academic bodies have published reporting guidelines (checklists of what developers should disclose), it has remained unclear to what extent commercial vendors adhere to these standards.
Methods
The researchers conducted a systematic audit of model documentation using a four-step process:
- Atomization: They analyzed 15 major reporting guidelines—including TRIPOD, Model Cards, and CONSORT-AI—and decomposed them into 220 unique "atoms," or individual pieces of requested information.
- Mapping: These atoms were categorized by model lifecycle stages, such as "Model Development," "Fairness," and "Prospective Evaluation."
- The Audit: The team reviewed the "Cognitive Computing Model Briefs" (the primary documentation provided to customers) for 12 of the most frequently used predictive models from Epic Systems, the dominant EHR vendor in the United States.
- Review and Adjudication: Four independent reviewers scored the presence of each atom. Discrepancies were resolved through a formal adjudication process to ensure a consensus-based final score.
Key Insights
The "Basics" Bias
The audit revealed that vendors are proficient at reporting "what" a model does but poor at explaining the "how" or "for whom." Documentation almost universally included the outcome definition and the Area Under the Receiver Operating Characteristic curve (AUROC). However, the actual model coefficients or weights—essential for transparency—were reported in only 8.3% of the briefs.
The Fairness and Reliability Gap
Despite the high stakes of algorithmic bias in medicine, information regarding fairness was missing from at least half of the documentation. Subgroup analyses by age, sex, or race/ethnicity were rarely performed or disclosed. Furthermore, reliability metrics were notably low: only 33% of models reported results from external validation, and 0% provided confidence intervals or statistical significance for their performance metrics.
Guideline Overload
With 220 unique items across 15 guidelines, the authors suggest that the sheer volume of recommendations may be "formidable" for developers. No single guideline currently covers the entire model lifecycle perfectly. For instance, while Model Cards excels at fairness reporting, TRIPOD provides better coverage of development details. The Model Facts Labels showed the highest adherence (74%), suggesting that simplified, standardized formats may be more effective than exhaustive checklists.
Results Summary
- Median Completion Rate: 39% of applicable reporting atoms were present.
- Guideline Adherence: The median adherence to individual guidelines was 54%.
- Transparency Issues: Only 8.3% of models provided saliency maps or coefficients.
- Clinical Utility: 0% of the documentation provided user-facing materials or specific directions for clinicians on how to interpret the model at the point of care.
Discussion and Future Work
The study's findings suggest that even with "generous credit"—such as counting a prompt to "contact a representative" as a reported strategy—commercial documentation falls short of professional standards. This lack of transparency prevents clinicians from identifying when a model might be out-of-scope or biased against specific patient demographics.
To bridge the AI chasm, the authors call for the creation of a "public dashboard" to track model adherence to reporting guidelines, similar to the tracking of clinical trials. Moving forward, professional societies must move beyond simply publishing guidelines and begin endorsing specific, operationalized reporting practices to incentivize vendors to move away from the "black box" approach.
Summary updated