Can Machine Learning Algorithms use Contextual Factors to Detect Unwarranted Clinical Variation from Electronic Health Record Encounter Data during the Treatment of Children Diagnosed with Acute Viral Pharyngitis

Apollo McOwiti, Yara Neaimeh, Jiaqi Gu, Yana Lalani, Taylor Newsome, Yvonne H nguyen, Sebastian Shrager, Laila Rasmy, Susan H. FentonmedRxiv 2026 · DOI

TL;DR

Researchers successfully used ensemble machine learning models to detect "unwarranted clinical variation" (UCV)—specifically inappropriate antibiotic prescribing for viral sore throats in children—by analyzing contextual factors like provider experience and clinic volume. The study demonstrates that ML can identify absolute deviations from medical guidelines using "weak labels" from electronic health records, potentially replacing labor-intensive manual chart reviews.

What Problem Does This Paper Address?

In medicine, Unwarranted Clinical Variation (UCV) occurs when patients with the same condition receive different treatments that aren't justified by their clinical needs or preferences. This leads to higher costs, potential harm, and a move away from evidence-based care.

The challenge is that traditional statistical methods (like mixed-effects regression) are great at measuring relative variation—how much Clinic A differs from Clinic B—but they struggle to measure absolute variation, or how much a specific treatment deviates from established guidelines. Furthermore, UCV is often driven by "Local Context Factors" (LCF)—the environment, the provider's background, and site-level dynamics—rather than the patient's biology. This paper investigates whether ML can leverage these contextual factors to flag inappropriate care automatically.

Methods

The authors framed UCV detection as a binary classification task: was the treatment warranted or unwarranted?

They employed three primary ensemble ML algorithms:

  • Random Forest (RF): A standard robust baseline.
  • CatBoost: Optimized for categorical data (like provider credentials or encounter types).
  • Explainable Boosting Machine (EBM): A "glass-box" model designed to provide both global and local feature importance, which is critical for gaining clinician trust.

Key Innovations: 1. Ontology Mapping: They used the UCVA Ontology to standardize features from Electronic Health Records (EHR), making the model more interoperable across different hospital systems. 2. Weak vs. Gold Labels: They compared models trained on "gold-standard" labels (painstakingly verified by human physicians via chart review) against "weak labels" (automated inferences directly from EHR data).

Datasets

The study utilized a retrospective dataset from the BIG-ARC clinical data warehouse at UTHealth Houston, standardized to the PCORnet Common Data Model.

  • Scope: Pediatric ambulatory visits (ages 3–19) between 2021 and 2024.
  • Focus: ICD-10 code J02.8 (acute pharyngitis due to other specified organisms), a proxy for viral sore throats where antibiotics are clinically inappropriate.
  • Size: 132 unique encounters across 24 clinics and 80 providers. While small for deep learning, it is a representative "real-world" slice for classical ML.

Key Insights and Ideas

  • The Case Volume Paradox: In surgical settings, higher volume usually correlates with better outcomes. However, this study found the opposite for ambulatory UCV: lower provider case volumes were associated with a reduced likelihood of inappropriate treatment. The authors suggest that providers with fewer patients might have more time to adhere strictly to guidelines rather than prescribing "preemptively" to move through a crowded waiting room.
  • Weak Labels are "Strong Enough": One of the most promising findings was that models trained on "weak labels" (automated EHR data) performed comparably to those trained on "gold labels" (manual review). This suggests we can scale UCV detection without needing thousands of hours of physician manual labor.
  • Context Trumps Clinicals: The model performed best when using only Local Context Factors (site/provider data). Adding patient-level clinical data didn't significantly improve results, reinforcing the theory that UCV is a systemic/provider issue, not a patient-driven one.
  • Experience Matters (But Not How You Think): Early-career providers and Nurse Practitioners (NPs) were actually less likely to prescribe unwarranted antibiotics than more experienced MDs, who might rely more on "clinical judgment" to override guidelines.

Key Results

  • High Predictive Accuracy: All three ensemble models (RF, CatBoost, EBM) achieved a median AUC of 0.91, significantly outperforming traditional logistic regression (AUC 0.85).
  • Top Predictors: Feature importance analysis identified site-level case volume, provider-level case volume, and provider credentials as the most influential factors in predicting unwarranted prescriptions.
  • Reliability: The final CatBoost model, evaluated on a held-out test set, maintained an AUC of 0.89 and a Brier score of 0.14, indicating excellent probability calibration.

Limitations and Future Work

  • Sample Size: The dataset was relatively small (n=132), which prevented the use of deep learning and may limit the generalizability of the specific "case volume" findings.
  • Localization: The study was conducted within a single academic health system. Because UCV is highly dependent on local culture, the model might need recalibration for different geographic regions.
  • Retrospective Bias: As with all EHR studies, if a provider didn't document a specific reason for a deviation (e.g., a phone call from a parent), the model might incorrectly flag the case as unwarranted.

The authors suggest that future work should integrate these models into real-time clinical dashboards to provide "nudge" feedback to providers, potentially reducing UCV as it happens.

Summary updated

Abstract

Abstract Rationale, Aims and Objectives Unwarranted clinical variation (UCV) in patient care often arises from contextual factors and contributes to increased costs, unnecessary treatments, and deviations from evidence-based practice. Detecting UCV is challenging due to the complexity of care decisions. Current approaches rely on centralized data aggregation and mixed-effects regression, which estimate relative variation but cannot detect absolute variation. Moreover, machine learning (ML) methods leveraging contextual factors for UCV detection are lacking. The objective is to demonstrate the feasibility of ML for identifying absolute UCV using contextual features extracted from electronic health records (EHR) and identify the factors correlated with UCV in treating acute viral pharyngitis in children. Methods We conducted a retrospective study of pediatric ambulatory visits (ICD-10 J02.8) at an academic health system. The use case focused on unwarranted antibiotic prescriptions for acute viral pharyngitis. We trained ensemble ML models—Random Forest, CatBoost, and Explainable Boosting Machine (EBM)—using encounter-level EHR data. Performance was evaluated using nested cross-validation and AUC metrics. We also compared CatBoost models trained on curated (gold-standard) versus weak labels. Results All three ML models demonstrated robust performance, with a median AUC of 0.91, using data from 24 clinics, 81 providers, and 122 patients within an academic health system. CatBoost models trained on weak labels exhibited performance comparable to those trained on gold-standard labels. Feature importance analysis indicated that site-level and provider-level case volumes were the most influential predictors, followed by provider credential, years of experience, and encounter type. Notably, lower provider case volumes were associated with a reduced likelihood of inappropriate treatment. Conclusions Classical ML models can effectively detect absolute UCV using contextual EHR features. Explainable models such as EBM offer interpretability critical for clinical adoption. These findings support ML-based approaches as scalable alternatives to traditional statistical methods for UCV detection without requiring centralized data analysis.
BibTeX
@misc{mcowiti2026can,
  author = {Apollo McOwiti and Yara Neaimeh and Jiaqi Gu and Yana Lalani and Taylor Newsome and Yvonne H nguyen and Sebastian Shrager and Laila Rasmy and Susan H. Fenton},
  title = {{Can Machine Learning Algorithms use Contextual Factors to Detect Unwarranted Clinical Variation from Electronic Health Record Encounter Data during the Treatment of Children Diagnosed with Acute Viral Pharyngitis}},
  year = {2026},
  doi = {10.64898/2026.02.23.26346757},
  howpublished = {medRxiv},
}
Figure