Developing a random forest algorithm to identify patent foramen ovale and atrial septal defects in Ontario administrative databases

Laura Oliva, Eric Horlick, Bo Wang, Ella Huszti, Ruth Hall, Lusine Abrahamyan

Introduction

In population health research, administrative databases are invaluable but often lack clinical granularity. A significant challenge in Ontario’s health records is the "coding collision" between Patent Foramen Ovale (PFO) and Atrial Septal Defect (ASD). Both conditions involve a hole in the heart's atrial septum and share identical ICD-9 (745.5) and ICD-10 (Q21.1) codes, yet they are clinically distinct. PFO is a common remnant of fetal circulation often linked to "cryptogenic" strokes, while ASD is a rarer congenital defect requiring different management. To enable precise long-term outcome research, this study developed a Random Forest (RF) machine learning model to distinguish between these two populations.

Methods

The researchers implemented a Random Forest classification algorithm using the randomForest package in R. RF was selected for its ability to handle high-dimensional data and complex variable interactions without requiring extensive a priori feature selection. The modeling process involved feature engineering of 24 variables, including demographics and specific intervention codes, and hyperparameter tuning to balance sensitivity and specificity. The RF model was benchmarked against a traditional heuristic method that classified patients based solely on prior stroke or TIA history.

Data and Cohort Selection

The study utilized data from ICES in Ontario, Canada, focusing on adult patients who underwent transcatheter closure for PFO or ASD between 2002 and 2017. To establish a "ground truth" for training, administrative records were linked to the CorHealth Ontario cardiac registry, which contains specific clinical fields distinguishing the two procedures.

Flowchart of patient selection for the classification cohort.

As shown in the selection process, the final cohort consisted of 1,482 patients, split between 697 PFO and 785 ASD cases.

Key Insights and Model Performance

The Random Forest model significantly outperformed traditional heuristics. While the traditional "stroke-history" proxy had high specificity (96%), its sensitivity was only 36%, missing the majority of PFO patients. In contrast, the tuned RF model achieved a more balanced performance with an accuracy of 0.76, sensitivity of 0.76, and specificity of 0.75.

Variable importance graph based on mean decrease in Gini index.

Analysis of variable importance revealed that patient age and the frequency of ischemic strokes prior to the procedure were the strongest predictors. PFO patients were generally younger and had a higher prevalence of prior stroke (39.5% vs. 3.7% for ASD). Additionally, intervention codes related to complex hemodynamic assessments, such as pulmonary artery pressure measurements, served as important proxies for identifying ASD patients.

Limitations and Future Directions

While successful, the model's performance is tied to the specific coding practices of Ontario's administrative databases, and external validation in other jurisdictions is needed. Furthermore, the model relies on the accuracy of underlying ICD codes for comorbidities like migraines. Future research will leverage this algorithm to evaluate long-term clinical outcomes, such as mortality and recurrent stroke, specifically comparing PFO and ASD populations across the province.

Summary updated

Abstract

Abstract Purpose: Routinely collected administrative data is widely used for population-based research. However, although clinically very different, atrial septal defects (ASD) and patent foramen ovale (PFO) share a single diagnostic code (ICD-9: 745.5, ICD-10: Q21.1). Using machine-learning based approaches we developed and validated an algorithm to differentiate between PFO and ASD patient populations within healthcare administrative data. Methods: Using data housed at ICES, we identified patients who underwent transcatheter closure in Ontario between October 2002 and December 2017 using a code (1HN80GPFL, N = 4680). A novel random forest model was developed using demographic and clinical information. Those patients who had undergone transcatheter closure and had records in the CorHealth Ontario cardiac procedure registry (N = 1482) were used as the reference standard. Several algorithms were tested and evaluated for accuracy, sensitivity, and specificity. Variable importance was examined via mean decrease in Gini index. Results: We tested 7 models in total. The final model included 24 variables, including demographic, comorbidity, and procedural information. After hyperparameter tuning, the final model achieved 0.76 accuracy, 0.76 sensitivity, and 0.75 specificity. Patient age group had the greatest influence on node impurity, and thus ranked highest in variable importance. Conclusions: Our random forest classification method achieved reasonable accuracy in identifying PFO and ASD closure in administrative data. The algorithm can now be applied to evaluate long term PFO and ASD closure outcomes in Ontario. Future external validation studies are recommended to further test the algorithm.
BibTeX
@article{oliva2022developing,
  author = {Laura Oliva and Eric Horlick and Bo Wang and Ella Huszti and Ruth Hall and Lusine Abrahamyan},
  title = {{Developing a random forest algorithm to identify patent foramen ovale and atrial septal defects in Ontario administrative databases}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2022},
  doi = {10.1186/s12911-022-01837-2},
}
Figure