Abstract
Purpose: Routinely collected administrative data is widely used for population-based research. However, although clinically very different, atrial septal defects (ASD) and patent foramen ovale (PFO) share a single diagnostic code (ICD-9: 745.5, ICD-10: Q21.1). Using machine-learning based approaches we developed and validated an algorithm to differentiate between PFO and ASD patient populations within healthcare administrative data. Methods: Using data housed at ICES, we identified patients who underwent transcatheter closure in Ontario between October 2002 and December 2017 using a code (1HN80GPFL, N = 4680). A novel random forest model was developed using demographic and clinical information. Those patients who had undergone transcatheter closure and had records in the CorHealth Ontario cardiac procedure registry (N = 1482) were used as the reference standard. Several algorithms were tested and evaluated for accuracy, sensitivity, and specificity. Variable importance was examined via mean decrease in Gini index. Results: We tested 7 models in total. The final model included 24 variables, including demographic, comorbidity, and procedural information. After hyperparameter tuning, the final model achieved 0.76 accuracy, 0.76 sensitivity, and 0.75 specificity. Patient age group had the greatest influence on node impurity, and thus ranked highest in variable importance. Conclusions: Our random forest classification method achieved reasonable accuracy in identifying PFO and ASD closure in administrative data. The algorithm can now be applied to evaluate long term PFO and ASD closure outcomes in Ontario. Future external validation studies are recommended to further test the algorithm.
Introduction
In the world of "Big Data" healthcare research, administrative databases are gold mines for population-level studies. However, they often suffer from a lack of granularity. A prime example is the distinction between Patent Foramen Ovale (PFO), a flap-like opening in the heart, and Atrial Septal Defects (ASD), a hole in the heart's wall. While clinically distinct, both conditions share identical diagnostic codes (ICD-9: 745.5, ICD-10: Q21.1) and procedural codes for transcatheter closure in Ontario.
This "coding collision" makes it nearly impossible for researchers to conduct accurate population-level studies on long-term outcomes for specific conditions. Previously, researchers attempted to separate these populations using simple heuristic logic—such as assuming a patient had a PFO if they had a recent stroke—but these methods were notoriously inaccurate and missed many cases. To solve this, researchers developed a machine learning algorithm to leverage complex clinical and demographic patterns.
Methods
The study employed a Random Forest (RF) classification approach. RF was chosen for its non-parametric nature, its ability to handle high-dimensional data, and its provision of implicit feature selection through "variable importance" metrics.
The researchers utilized a sophisticated linkage of several Ontario-based datasets housed at ICES, including hospital visits and physician billings. The CorHealth Ontario Cardiac Registry served as the "ground truth" reference standard, as it contains specific clinical fields indicating whether a procedure was for a PFO or an ASD. The study cohort included 1,482 adult patients who underwent transcatheter closure between 2002 and 2017 and had records in both ICES and the CorHealth registry.
Seven different model iterations were tested using the randomForest package in R, eventually settling on a version with 24 variables. The data was split into training (40%) and test (60%) sets, and hyperparameters were tuned to balance sensitivity and specificity.
Results and Key Insights
The final tuned Random Forest model significantly outperformed traditional heuristic methods. It achieved an accuracy of 0.76, a sensitivity of 0.75, and a specificity of 0.76. While traditional methods were highly specific (0.96), they suffered from a dismal sensitivity (0.36). By sacrificing a small amount of specificity, the RF model effectively doubled the sensitivity, allowing for a much more representative identification of the PFO population.
Variable importance analysis revealed that patient age was the most influential factor. ASD patients in the cohort tended to be older and more likely to be female compared to PFO patients.
Furthermore, the model demonstrated that relying on stroke history alone is insufficient. By combining stroke data with other comorbidities (like migraine or heart failure) and specific intervention codes, the predictive power was significantly boosted. For instance, specific procedural codes related to heart capacity and pressure measurements were key discriminators between the two groups.
Discussion and Conclusion
This research provides a robust solution to the coding overlap problem in Ontario's administrative health data. The Random Forest algorithm offers a balanced performance that is far superior to previous heuristic approaches, enabling more accurate population-level research into the long-term outcomes of PFO and ASD closures.
However, limitations remain. The model's performance is dependent on the quality of the underlying ICD codes, and its generalizability to other jurisdictions with different coding practices is currently unknown. Future work will focus on applying this algorithm to the broader ICES database to conduct long-term outcome studies and pursuing external validation in other healthcare systems. This machine learning approach represents a significant step forward in extracting granular clinical insights from large-scale administrative datasets.
Summary updated