TL;DR
This paper tackles the time-consuming manual chart review process for identifying postoperative complications by developing automated machine learning models using structured Electronic Health Record (EHR) data. They found that a multi-task learning approach called Propensity Weighted Observations (PWO) significantly outperformed other methods, demonstrating the potential to accelerate clinical research and quality improvement.
What Problem Does This Paper Address?
Manual Chart Review (MCR) is a cornerstone of clinical research and quality improvement initiatives, such as those by the American College of Surgeons (ACS) and the Centers for Disease Control and Prevention (CDC). However, MCR is incredibly labor-intensive and time-consuming, creating a bottleneck for extracting crucial patient outcome data. This is particularly true for identifying postoperative complications like surgical site infections (SSIs), pneumonia, urinary tract infections (UTIs), sepsis, and septic shock, which are significant drivers of healthcare costs and patient mortality. The paper aims to automate this process by leveraging structured EHR data, thereby accelerating data extraction and reducing the burden on healthcare professionals.
Methods
The researchers explored six different machine learning approaches to detect postoperative complications:
- Single-Task Learning: This served as a baseline, where an independent model was trained for each specific complication.
- Multi-Task Learning (MTL) Methods: These methods leverage the inherent relationships between different complications to improve model performance. The paper investigated five MTL approaches:
- Hierarchical Classification: Models are built in a top-down fashion, starting with detecting any complication, then distinguishing between types, and finally identifying subtypes (e.g., SSI subtypes).
- Offset Method: Similar to hierarchical, but child models incorporate predictions from parent models as an offset term, encouraging similarity.
- Propensity Weighted Observations (PWO): This method builds classifiers from the top down but weights patients based on their propensity to have a complication, giving more influence to those more likely to be affected. This is a key innovation.
- Multi-Task Learning with Penalties (MTLP): This approach learns all task coefficients simultaneously, penalizing differences between parent and child task models to enforce similarity.
- Partial Least Squares (PLS): This method models all tasks simultaneously but autonomously learns the relationships between outcomes without explicit expert input on the hierarchy.
The models were built using Lasso-penalized logistic regression to handle variable selection and small sample sizes.
Datasets
The study utilized data from surgical patients at the University of Minnesota Medical Center collected between 2011 and 2014, specifically those included in the ACS-NSQIP registry. The data was split into a training set (first 2.5 years) for model development and a test set (last 1.5 years) for evaluation. The dataset included demographic information, laboratory results, microbiological results, diagnosis codes, orders, procedures, vital signs, and medications. The gold standard for evaluation was the manual chart review data collected by NSQIP reviewers.
Key Insights and Ideas
- The Power of Hierarchy: The paper strongly supports the idea that complications are not independent events and can be organized hierarchically. Multi-task learning methods that exploit this hierarchy generally outperform single-task learning.
- PWO's Novelty: The Propensity Weighted Observations (PWO) method, a multi-task learning approach, emerged as the top performer. Its innovative use of propensity weighting to enrich the training sample with relevant patients, rather than strictly excluding others, proved highly effective. This approach avoids the pitfalls of hierarchical methods that might exclude patients prematurely based on early predictions.
- MTL for Rare Events: Multi-task learning methods, particularly MTLP and PWO, showed promise in detecting rare complications (like septic shock) where single-task learning might struggle due to small sample sizes.
- EHR Data is Rich: Structured EHR data contains significant indicators for postoperative complications, making automated detection feasible and accurate.
Key Results
- High Detection Performance: All six methods demonstrated good detection performance, with Area Under the Curve (AUC) values exceeding 0.8 for all complications and even 0.9 for some.
- PWO Outperforms: The PWO method achieved the statistically significant highest performance overall.
- Single-Task and MTLP Close Seconds: Single-task learning and MTLP performed very similarly and were close behind PWO.
- PLS Lagged: The Partial Least Squares (PLS) method showed the worst overall performance, likely due to its inability to infer complex relationships correctly with limited data.
- Specific Strengths: PWO was particularly strong in detecting deep SSI and overall SSI. Single-task, PWO, and MTLP were top performers for sepsis detection.
Limitations and Future Work
- Limited to Structured Data: This study focused solely on structured EHR data. The authors acknowledge that combining structured and unstructured data (like physician narratives) could further improve detection performance.
- Sample Size for PLS: The authors suggest that the relatively small sample sizes for some rare complications might have hindered PLS's ability to infer relationships correctly. With substantially more data, PLS might perform better.
- Overfitting in MTLP: The MTLP method showed signs of overfitting, with significantly higher AUC on the training set compared to the test set for some outcomes.
- Future Work: The authors plan to investigate models that combine both structured and unstructured EHR data to enhance detection capabilities. They also aim to compare the performance of these models against manual chart review to quantify the acceleration achieved.
Summary updated