Abstract
Objective: To determine whether clinicians will use machine learned clinical order recommender systems for electronic order entry for simulated inpatient cases, and whether such recommendations impact the clinical appropriateness of the orders being placed.
Materials and Methods: 43 physicians used a clinical order entry interface for five simulated medical cases, with each physician-case randomized whether to have access to a previously-developed clinical order recommendation system. A panel of clinicians determined whether orders placed were clinically appropriate. The primary outcome was the difference in clinical appropriateness scores of orders for cases randomized to the recommender system. Secondary outcomes included usage metrics and physician opinions.
Results: Clinical appropriateness scores for orders were comparable for cases randomized to the recommender system (mean difference -0.1 order per score, 95% CI:[-0.4, 0.2]). Physicians using the recommender placed more orders (mean 17.3 vs. 15.7 orders; incidence ratio 1.09, 95% CI:[1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Approximately 95% of participants agreed the system would be useful for their workflows.
Conclusions: Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.
Introduction
Modern medicine is increasingly data-driven, yet physician adherence to evidence-based guidelines remains inconsistent, often hovering between 20% and 80%. Traditional Clinical Decision Support (CDS) tools, such as static "order sets," are typically developed through committee-based "top-down" approaches. These tools can be rigid and fail to keep pace with evolving clinical workflows, contributing to the "click fatigue" prevalent in Electronic Health Record (EHR) systems. This study explores a "bottom-up" alternative: a machine-learning recommender system designed to predict and suggest clinical orders based on historical patient data.
Methods and System Design
Researchers at Stanford University developed a simulated Computerized Physician Order Entry (CPOE) interface powered by a collaborative filtering algorithm—the same technology behind consumer recommendation engines like Amazon or Netflix. The system was trained on the STRIDE clinical data warehouse, encompassing over 11 million order instances (medications, labs, imaging) from 74,000 patients between 2009 and 2014.
The study involved 43 physicians (residents and faculty) who completed five simulated clinical cases, including bacterial meningitis and pulmonary embolism. Participants were randomized to either a control group (manual search/standard order sets) or an intervention group with access to the recommender system.
To ensure safety and quality, a panel of board-certified clinicians used a Delphi method to grade every possible order on a scale from -10 (harmful) to +10 (highly beneficial).
Key Results
Clinical Impact and Appropriateness
The study found no significant difference in the quality of care between the two groups. The mean difference in appropriateness scores was negligible (-0.1 on a 20-point scale), indicating that the machine-learning system did not lead doctors toward dangerous or inappropriate clinical paths.
Order Volume and Workflow Efficiency
Physicians using the recommender placed slightly more orders on average (17.3 vs. 15.7). This suggests a "nudge" effect, where the tool helps doctors remember beneficial but non-essential orders they might have otherwise overlooked. Crucially, users in the recommender group experienced 10% fewer navigation clicks, reducing the cognitive load of searching through complex menus.
User Experience and Sentiment
Physician acceptance was remarkably high:
- 100% of physicians with access to the tool chose to use it.
- 95% agreed the system would be useful in their real-world workflow.
- 90% agreed the system would make their job easier.
- 86% felt it would increase their productivity.
The recommender system also demonstrated superior "recall" (58% vs. 42% for manual search), meaning the algorithm was more effective at surfacing the specific orders doctors actually needed than the doctors were at finding them via keyword search.
Discussion and Future Work
The results demonstrate that machine-learned clinical order recommendations can be successfully integrated into physician workflows without compromising clinical appropriateness. By providing relevant suggestions based on large-scale historical data, these systems can meet physician needs more effectively than traditional keyword-based searches.
While the study was limited by its simulated environment, the high level of physician acceptance and the reduction in navigation clicks point toward a promising future for data-driven CDS. Future research will focus on implementing these systems in "live" clinical settings to evaluate their impact on real-world physician burnout and patient outcomes in high-stress environments like the ER or ICU.
Summary updated