Learning to Identify Severe Maternal Morbidity from Electronic Health Records

Cheng Gao, Sarah Osmundson, Xiaowei Yan, Digna Velez Edwards, Bradley A. Malin, You Chen

TL;DR

This paper introduces a machine learning framework to automatically identify Severe Maternal Morbidity (SMM) from electronic health records (EHRs), outperforming existing methods like the CDC criteria. The framework not only improves identification accuracy but also uncovers novel indicators of SMM, offering a more scalable and efficient approach to monitoring maternal health quality.

What Problem Does This Paper Address?

Severe Maternal Morbidity (SMM) refers to significant complications during pregnancy that negatively impact a woman's health. Identifying SMM is crucial for improving maternal healthcare quality, as it can serve as an indicator of near misses for maternal mortality. However, current methods for identifying SMM face several challenges:

  • Lack of Consensus Definition: There isn't a universally agreed-upon definition of SMM, leading to variations in how it's identified.
  • Limited Scalability: Existing definitions, like the CDC's, often rely on diagnosis and procedure codes but can lead to many false positives and negatives because they don't fully capture the severity. Other guidelines, like those from ACOG/SMFM, require manual chart review, which is time-consuming and labor-intensive, and their facility-specific nature limits generalizability.
  • Difficulty in Identifying Risk Factors: Understanding the underlying risk factors and contributing causes of SMM is essential for prevention and management, but this is challenging with manual review processes.

This paper aims to address these issues by developing an automated, scalable, and more accurate method for SMM identification using machine learning on EHR data.

Methods

The authors developed a machine learning framework for SMM identification. The core of their approach involves:

  • Feature Engineering: Medical concepts (diagnoses and procedures) from EHRs are extracted and transformed into binary features. These concepts are standardized using the OMOP common data model, promoting interoperability across different institutions.
  • Feature Selection: To handle the large number of medical concepts, a filter feature selection algorithm is used. This algorithm ranks concepts by their frequency in the patient population, and the top-ranking features are incrementally added to the machine learning models. This approach aims for generalizability by not relying on class information during initial selection.
  • Machine Learning Models: The study primarily uses ridge logistic regression (L2 regularization) as the machine learning model. They train and evaluate four models:
  • M₀ (Baseline): The CDC criteria model, serving as a benchmark.
  • M₁: Uses diagnosis concepts as features.
  • M₂: Uses procedure concepts as features.
  • M₃: Uses both diagnosis and procedure concepts as features.
  • Handling Imbalanced Data: Given that SMM is rare (less than 1% of deliveries), the dataset is highly imbalanced. To mitigate bias, the authors investigate different case:control ratios (1:1, 1:4, 1:16, 1:64) during model evaluation.
  • Performance Evaluation: Models are evaluated using standard metrics: Sensitivity (SN), Specificity (SP), Positive Predictive Value (PPV), and Area Under the Receiver Operating Characteristic Curve (AUC).
  • Risk Factor Identification: A medical concept ranking algorithm is employed to identify important risk factors, considering both the concept's importance within a model (coefficient) and the model's overall performance (AUC).

Datasets

The study utilized a dataset from the Vanderbilt University Medical Center (VUMC) EHR system, comprising 45,858 deliveries from 2005 to 2018. This dataset included patient demographics, admission notes, discharge summaries, and clinical concepts (diagnoses and procedures). The diagnosis and procedure codes were transformed into the OMOP common data model.

The authors constructed a cohort of SMM cases based on ACOG criteria (four or more units of blood transfusion or ICU admission), which were then manually reviewed by obstetricians to exclude non-SMM cases. This resulted in 412 SMM cases (0.9% of deliveries), with the remaining deliveries serving as controls.

Key Insights and Ideas

  • Machine Learning Outperforms Traditional Criteria: The developed machine learning framework significantly outperforms the baseline CDC criteria in identifying SMM, demonstrating the power of ML in this domain.
  • Scalability and Generalizability: By using standardized clinical terminologies (OMOP), the framework is designed to be applicable to EHRs from other institutions, addressing a key limitation of current methods.
  • Discovery of Novel SMM Indicators: The study identified several "novel" concepts (disorders of fluid or electrolytes, systemic inflammatory response syndrome, and acidosis) that were strongly associated with SMM and not previously included in common SMM definitions. This suggests new avenues for SMM detection and understanding.
  • Feature Importance Matters: The performance of the ML models improved as more features were included, but there was a point of diminishing returns, highlighting the importance of selecting the right set of features.
  • Case:Control Ratio Impact: The optimal case:control ratio for model performance varied depending on the model, indicating that careful consideration of data balancing is necessary for robust results.

Key Results

  • The machine learning framework achieved a significantly higher AUC of 0.94 compared to the CDC model's AUC of 0.80.
  • The framework improved Positive Predictive Value (PPV) by 59% (from 0.22 for CDC to 0.35 for the proposed model).
  • Model M₃ (using both diagnoses and procedures) with 80% of features achieved the best AUC of 0.937, which was 18.6% better than the baseline Mo.
  • Specificity improved by about 1% over the baseline model M₀ at 10% of features for M₁.
  • Sensitivity improved by 25% from 0.614 for Mo to 0.765 for M₃ (when 80% of features were used).
  • The PPV clearly improved across models M₁, M₂, and M₃, with M₂ achieving the best PPV of 0.347 (at 80% features), which was 57% better than the baseline Mo.
  • Among the top 10 important concepts, three novel risk factors were identified: disorders of fluid and electrolytes (85% of patients with this concept experienced SMM), systemic inflammatory response syndrome (82%), and acidosis.

Limitations and Future Work

The authors acknowledge several limitations:

  • Control Group Assumption: The study assumed that the control group did not contain any SMM cases, which is a challenge to guarantee without extensive manual review of all control instances. However, they note that the small missed-case:control ratio likely has minimal impact.
  • Limited ML Algorithms: Only logistic regression was used. Future work will explore more advanced algorithms like neural networks, decision trees, SVMs, and gradient boosting classifiers.
  • Feature Selection: The current feature selection is based on frequency and doesn't consider correlations between features. More advanced methods could improve performance by removing duplicated features.
  • Feature Scope: The study only considered diagnosis and procedural features, neglecting potentially important features like laboratory variables, medications, and vital signs.
  • Atemporal Concept Assignment: Concepts were considered in an atemporal fashion. Modeling temporal relationships between concepts could enhance identification performance and provide insights into SMM development and management.

Summary updated

Abstract

Severe maternal morbidity (SMM) is broadly defined as significant complications in pregnancy that have an adverse effect on women’s health. Identifying women who experience SMM and reviewing their obstetric care can assist healthcare organizations in recognizing risk factors and best practices for management. Various definitions of SMM have been posited, but there is no consensus. Existing definitions are further limited in that they 1) are often rooted in existing clinical knowledge (which is problematic as many risk factors remain unknown), leading to poor positive predictive performance (PPV), and 2) have limited scalability as they often require substantial chart review. Thus, in this paper, a machine learning framework was introduced to automatically identify SMM and relevant risk factors from electronic health records (EHRs). We evaluated this framework with EHR data from 45,858 deliveries at a large academic medical center. The framework outperformed a state-of-the-art model from the U.S. Centers for Disease Control and Prevention (AUC of 0.94 vs. 0.80). Specially, it improved upon PPV by 59% (CDC: 0.22 vs. our model: 0.35). In the process, we revealed several novel SMM indicators, including disorders of fluid or electrolytes, systemic inflammatory response syndrome, and acidosis.
BibTeX
@inproceedings{gao2019learning,
  author = {Cheng Gao and Sarah Osmundson and Xiaowei Yan and Digna Velez Edwards and Bradley A. Malin and You Chen},
  title = {{Learning to Identify Severe Maternal Morbidity from Electronic Health Records}},
  booktitle = {MedInfo},
  year = {2019},
  doi = {10.3233/shti190200},
}
Figure