TL;DR
This paper explores how to quantify the similarity between patient medical records using SNOMED CT, a medical ontology. While their ontology-based metrics showed some improvement over basic methods, they still fell short of human expert judgment, suggesting that while ontologies are valuable, more sophisticated approaches are needed for truly expert-level patient similarity assessment.
What Problem Does This Paper Address?
In the era of electronic health records (EHRs), vast amounts of patient data are becoming available. Beyond direct patient care, this data holds immense potential for clinical research, quality improvement, and decision support. A key challenge in unlocking this potential is the ability to efficiently identify similar patients. This is crucial for tasks like finding suitable candidates for clinical trials, understanding disease progression across patient cohorts, or identifying best practices. Traditional methods often struggle with the complexity and richness of medical data, leading researchers to explore how structured medical terminologies and ontologies, like SNOMED CT, can provide a more robust foundation for measuring patient similarity.
Methods
The authors developed and evaluated five different "inter-patient distance metrics" to quantify similarity between patient records.
- Metric 1 (Bag of Findings): This is a knowledge-free baseline that treats each patient's record as an unstructured collection of findings and calculates similarity based on the proportion of findings present in one record but not the other.
- Metrics 2-5 (SNOMED CT Based): These metrics leverage the SNOMED CT ontology, which organizes medical concepts and their relationships (like "is-a" or "part-of"). They explore different ways to use the "distance" between SNOMED CT concepts within patient records to derive an overall patient similarity score.
- Metric 2: Calculates the average minimal number of links between concepts in two patient records.
- Metric 3: Similar to Metric 2 but weights the links using information content derived from the number of descendants of a concept in the SNOMED CT hierarchy.
- Metric 4: Also uses information content for weighting but derives it from term frequency within a corpus, aiming to be less sensitive to SNOMED CT's hierarchical structure.
- Metric 5: This is the most complex, combining information content for weighting with a "clinical distance" that considers the path between concepts and accounts for the concept's informativeness (e.g., specific diagnoses vs. broader categories).
Datasets
The study utilized a dataset of 30 random patient cases from the Columbia University Medical Center (CUMC) clinical data repository, spanning from 1989 to 2003. This data included discharge summaries, clinic notes, and other text-based clinical documents, as well as coded administrative data (ICD9-CM).
Key Insights and Ideas
- Ontologies Offer an Edge, But Not a Panacea: While using SNOMED CT and information content principles improved similarity metrics compared to a simple "bag of findings" approach, the gains were modest. This suggests that while structured knowledge is beneficial, it doesn't automatically bridge the gap to expert-level understanding.
- Expert Judgment is the Gold Standard: The study highlights the significant gap between automated metrics and human expert assessment. Expert inter-rater reliability was very high (0.91), indicating a consistent understanding of similarity among physicians.
- "Clinical Finding" is Key: When analyzing similarity along specific SNOMED CT axes, "Clinical Finding," "Body Structure," and "Procedure" were the most predictive of expert similarity judgments. This implies that focusing on these core aspects of a patient's condition is crucial for capturing clinical similarity.
- Complexity Doesn't Guarantee Improvement: Surprisingly, the more complex metrics (3-5) did not consistently outperform the simpler SNOMED CT-based metric (Metric 2). This suggests that simply adding more sophisticated ontology features doesn't automatically lead to better results.
Key Results
- Expert inter-rater reliability for similarity judgments was 0.91.
- The correlation between expert similarity scores and the automated metrics ranged from 0.27 (for the "bag of findings" metric) to 0.30 (for the more complex SNOMED CT-based metrics).
- When focusing solely on the "Clinical Finding" axis of SNOMED CT, the correlation increased to 0.37.
- Experts were statistically more likely to agree with each other than any of the algorithms were to agree with the experts (p < 0.0001).
Limitations and Future Work
- Data Granularity and Conversion Issues: The authors acknowledge that converting data from source terminologies (like ICD9-CM) to SNOMED CT may have introduced inaccuracies and loss of granularity, potentially impacting metric performance. They suggest that using data initially encoded in SNOMED CT could yield better results.
- Ontology Quality and Coverage: The effectiveness of ontology-based metrics is highly dependent on the quality and completeness of the ontology itself. The authors speculate that certain domains within SNOMED CT might be better developed than others.
- Understanding Expert Cognition: A significant area for future work is to formally study the cognitive processes experts use when assessing patient similarity. Understanding how experts arrive at their judgments could inform the development of more effective automated metrics.
- Exploring Other Terminologies: While SNOMED CT was chosen for its strengths, comparing its performance to other terminologies like ICD9-CM in a follow-up experiment is suggested.
- Focusing on High-Yield Data: Given that experts primarily relied on discharge summaries, future work could explore applying these metrics to specific document types to see if performance improves.
Summary updated