TL;DR
This paper introduces an automated method to compare healthcare processes for different patient subpopulations, using event data from electronic health records. By converting discovered process models into graphs, they enable quantitative comparison, which is crucial for quality control and understanding variations in patient care, especially in complex areas like cancer treatment.
What Problem Does This Paper Address?
Hospitals, like any complex organization, often see patient care processes deviate from intended standards due to unforeseen events and inherent complexity. For effective quality control and improvement, it's vital to understand how these processes differ across various patient groups (e.g., different cancer types or severity levels). While process mining can uncover these "as-is" processes from electronic health records (EHRs), manually comparing the resulting complex process models for numerous subpopulations is time-consuming and impractical. This paper aims to automate this comparison process, providing a quantitative measure of similarity between these patient care pathways.
Methods
The core of the approach involves several key steps:
- Process Discovery: EHR data is transformed into event logs, which are then used to discover process models. The authors primarily use the Inductive Visual Miner (IvM) for this, which also helps identify deviations from the norm. For the MIMIC dataset, Petri nets were used due to software issues.
- Graph Construction: The discovered process models (Petri nets or IvM models) are converted into directed graphs. This step is crucial for enabling graph-based similarity calculations.
- Process Model Comparison: Three main methods are employed:
- Visual Inspection: A user study with 20 participants was conducted to establish a "ground truth" for visual similarity between pairs of process models, using a Likert scale.
- Cross-Log Conformance Checking: This involves replaying the event log of one patient subpopulation against the process model discovered for another subpopulation (and vice-versa). Metrics like fitness, precision, and generalization are used.
- Graph-Based Similarity: Standard graph similarity measures are applied to the constructed directed graphs. This includes:
- Graph Edit Distance (GED): Measures the minimum number of operations (node/edge additions, deletions, substitutions) to transform one graph into another. An approximation algorithm was used due to GED's computational complexity.
- Feature-Based Similarity (FS): A feature vector is extracted for each graph (e.g., number of nodes, edges, average degree), and similarity is calculated based on the Euclidean distance between normalized feature vectors.
Datasets
The study utilizes two distinct datasets:
- MIMIC-III Database: A large, publicly available critical care database from Beth Israel Deaconess Medical Center. The authors focus on different cancer types within this dataset, using event data to model patient pathways in intensive care.
- Hospital Group Twente (ZGT) Database: A private database from a Dutch hospital, specifically used to analyze breast cancer patient care pathways, differentiating between various patient subpopulations based on factors like BI-RADS scores and age.
Key Insights and Ideas
- Automating Complexity: The primary insight is that manual comparison of complex healthcare process models is infeasible. Developing automated methods is essential for scaling quality improvement efforts.
- Graph Representation is Key: Converting process models into directed graphs unlocks a wide range of established graph similarity algorithms, making quantitative comparison possible.
- Human Judgment is a Challenging Ground Truth: The user study highlights that even for trained individuals, assessing process model similarity is subjective and can have high variance, underscoring the need for objective, automated metrics.
- Context Matters for Similarity Metrics: The effectiveness of different similarity measures (fitness, precision, GED, FS) varies significantly between the MIMIC (cancer types) and ZGT (breast cancer) datasets. This suggests there isn't a one-size-fits-all solution for process model comparison.
Key Results
- MIMIC Dataset: For cancer types in intensive care, average fitness was found to be a good indicator of visual similarity.
- ZGT Dataset: For breast cancer patients, average precision and Graph Edit Distance (GED) showed a strong correlation with human visual impression of similarity.
- Feature-Based Similarity (FS): The feature-based similarity measure (FS) generally showed high values (above 0.9) and little variation across comparisons in both datasets, making it less useful for distinguishing subtle differences in process models.
- GED Approximation: While GED showed promise, the authors note that their approximation algorithm might not always yield the true GED, especially in cases where it should be zero.
Limitations and Future Work
- Limited Data Points: The study's conclusions are based on a relatively small number of pairwise comparisons (6 for MIMIC, 6 for ZGT), limiting the generalizability of the findings.
- Approximation of GED: The use of an approximate GED algorithm means the results might not always be exact. Future work could focus on improving the computational efficiency of GED calculation.
- Domain-Specific Results: The differing results between the MIMIC and ZGT datasets suggest that the "best" similarity metric is context-dependent. Further research is needed to determine which metrics or combinations are most suitable for different types of process models and application domains.
- Qualitative Aspects: The current graph-based measures primarily focus on structure. Future work could explore semantic embeddings to capture richer meaning in graph comparison.
- Human Ground Truth: Investigating the validity of human ground truth using artificially constructed process models in a more general domain is suggested to better understand the reliability of human judgments.
Summary updated