TL;DR
This paper explores the use of machine learning (ML) to predict antibiotic resistance in Enterobacterales bloodstream infections, aiming to improve empirical treatment decisions. While both clinicians and ML models face challenges, the study suggests that ML could potentially increase the proportion of patients receiving effective treatment by up to 9% compared to current clinical practice, a significant improvement in the context of antimicrobial stewardship.
What Problem Does This Paper Address?
Severe bacterial infections, particularly those caused by Gram-negative bacteria like Enterobacterales, require prompt antibiotic treatment to prevent serious adverse outcomes. However, definitive identification of the causative pathogen and its antibiotic resistance profile typically takes 24-48 hours. This delay necessitates empirical treatment based on clinical judgment and general guidelines, which can lead to either under-treatment (if resistance is present and not accounted for) or over-treatment (using broad-spectrum antibiotics unnecessarily, driving antimicrobial resistance). The challenge lies in accurately predicting resistance before definitive lab results are available to guide optimal, timely, and stewardship-conscious antibiotic choices.
Methods
The researchers employed XGBoost, a gradient boosting algorithm, to build predictive models for antibiotic resistance. They trained models using a comprehensive set of features derived from electronic health records (EHRs) and community microbiology data. Key features included patient demographics, comorbidities, previous antibiotic exposure, clinical syndrome, and historical laboratory and microbiology results.
Two main modeling approaches were used:
- Baseline Model: Predictions were made using data available at the time of blood culture collection, before species identification.
- Species Identification Model: Predictions were made after the bacterial species was identified, incorporating this information as an additional input.
Model performance was evaluated using standard metrics like the Area Under the Receiver Operating Characteristic Curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). They also compared the ML model's predictions to actual clinician prescribing practices, analyzing outcomes in terms of optimal, under-, and over-treatment.
Datasets
The study utilized de-identified data from the Infections in Oxfordshire Research Database, encompassing patients from Oxford University Hospitals (OUH) in the UK.
- Training Data: Blood cultures collected between January 1, 2017, and December 31, 2021.
- Test Dataset 1: Blood cultures collected between January 1, 2022, and December 31, 2022.
- Test Dataset 2: Blood cultures collected between January 1, 2023, and December 31, 2023.
This dataset included 4709 Enterobacterales bloodstream infections from 4243 patients. The study focused on resistance to seven common antibiotics: amoxicillin, co-amoxiclav, ceftriaxone, piperacillin-tazobactam, ciprofloxacin, co-trimoxazole, and gentamicin.
Key Insights and Ideas
- Species Identification Matters: Incorporating bacterial species identification significantly improved the predictive performance of the ML models for most antibiotics. This highlights the value of this intermediate laboratory result in refining predictions.
- Past Resistance is a Strong Predictor: The most influential feature for predicting resistance across many antibiotics was the time since the last isolate with resistance to that specific antibiotic. This underscores the importance of historical resistance patterns in guiding current treatment.
- Clinician vs. Model Performance: While predicting resistance is challenging for both clinicians and ML models, the study demonstrates that ML models, even with modest performance, have the potential to improve upon current clinical practice by increasing the proportion of patients receiving active and optimally targeted treatment.
- The Stewardship Dilemma: The study grapples with the balance between ensuring active treatment and minimizing the use of broad-spectrum antibiotics. ML models offer a way to potentially navigate this by identifying patients who truly need broader coverage versus those who can be treated with narrower-spectrum agents.
Key Results
- Model Performance: In held-out test data, AUCs for predicting resistance ranged from 0.680 to 0.737 at the time of blood culture sampling. When species identification was included, AUCs improved, reaching up to 0.827 for amoxicillin.
- Clinician Treatment Outcomes: In the clinical comparison, 70% of patients received an active beta-lactam. However, 44% were over-treated (broader spectrum than needed), 26% were optimally treated, and 30% were under-treated.
- ML Model Impact: Without species data, ML predictions could have led to 79% of patients receiving active treatment, with 45% over-treated, 34% optimally-treated, and 21% under-treated. This represents a potential increase of up to 9% in optimally treated patients compared to clinicians.
- Model Stability: Model performance remained relatively stable over time, and retraining or incremental updating of models was quick and efficient.
Limitations and Future Work
- Data Limitations: The study acknowledges that the models are trained on patients with positive blood cultures, and the predictors of resistance might differ for patients without positive cultures. The focus on Enterobacterales also excludes other pathogens and polymicrobial infections.
- Community Data: The absence of data on community antibiotic exposures is noted as a potential limitation, as this information could have further improved model performance.
- Allergies: Patient allergy data was not available, which could impact the real-world applicability of certain model-driven treatment switches (e.g., to ceftriaxone).
- External Validation: While the models were validated on internal datasets, external validation in different healthcare settings is crucial before deployment.
- Future Directions: The authors suggest that future work could focus on incorporating community prescribing data, expanding models to include other pathogens, and conducting external validation. They also highlight the need for better models for narrow-spectrum agents like amoxicillin and co-amoxiclav.
Summary updated