TL;DR
This paper tackles the problem of redundant information in electronic health records (EHRs) by developing and evaluating automated methods to identify clinically relevant new information in patient notes. They found that while semantic similarity measures didn't significantly improve detection accuracy over baseline language models, they did boost recall, which is crucial for not missing vital patient data. The study also revealed significant variations in note redundancy across different medical specialties.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) offer immense potential for improving patient care and research, but they also introduce challenges. A major issue is the pervasive use of "copy-paste" functionality, leading to a significant amount of redundant and sometimes outdated information within clinical notes. This redundancy not only inflates note length but also makes it difficult for clinicians and researchers to quickly identify truly new and clinically relevant information. This can lead to errors, communication breakdowns, and increased cognitive load when synthesizing patient histories, ultimately hindering efficient patient care and data analysis. The paper aims to automate the detection of this "new" information to alleviate these problems.
Methods
The researchers employed a multi-pronged approach:
- Statistical Language Models (SLMs): They built upon existing n-gram language models, specifically focusing on bigram models, which had previously shown good performance. These models assign probabilities to sequences of words, helping to understand the structure and content of notes.
- Semantic Similarity Measures: To go beyond simple word matching, they incorporated three information content (IC)-based semantic similarity measures: Resnik, Jiang and Conrath, and Lin. These metrics aim to quantify the similarity between biomedical concepts, even if they are expressed using different terms (e.g., "heart" vs. "cardiac").
- Heuristic Rules: Developed rules were used to classify clinical relevance, leveraging information like section headers and clinical note content. For instance, vital signs were generally considered relevant new information.
- Gold Standard Creation: A crucial step involved creating a manually annotated "gold standard" dataset. Four medical students annotated clinical notes for new information, and a resident physician reviewed these annotations to ensure high quality and resolve disagreements.
- Comparison and Evaluation: The performance of various automated methods (baseline SLM vs. SLM with semantic similarity) was evaluated against this gold standard using metrics like precision, recall, and F-measure.
Datasets
- Training and Evaluation Corpus: A corpus of 591 progress notes from 40 inpatient admissions was used to develop and evaluate the automated methods. This dataset was manually annotated by physicians to create a reference standard.
- Redundancy Analysis Corpus: A larger dataset comprising 71,021 outpatient notes and 64,695 inpatient notes from 500 solid organ transplant patients (collected between April and August 2015) was used to quantify and compare redundancy across different specialties and settings.
Key Insights and Ideas
- Recall is King for Clinical Relevance: While semantic similarity didn't boost overall accuracy (F-measure), it significantly improved recall. This is critical in a clinical setting, as missing a piece of new, relevant information can have serious consequences.
- Specialty Matters for Redundancy: The study revealed that note redundancy isn't uniform across all medical fields. Pediatric and Psychiatry notes, for example, showed higher redundancy than Radiology or Surgery notes. This suggests that interventions to reduce redundancy might need to be tailored to specific specialties.
- Semantic Similarity's Limited Impact: The authors found that semantic similarity measures, while conceptually appealing, didn't offer a substantial performance boost over simpler language models for detecting new information. This might be because clinicians tend to copy/paste entire sentences or rephrase concepts rather than subtly changing wording, limiting the utility of fine-grained semantic comparisons.
Key Results
- The best-performing method achieved a recall of 0.87, precision of 0.62, and an F-measure of 0.72.
- Adding semantic similarity measures improved recall by approximately 2% compared to the baseline model, but did not significantly alter precision or F-measure.
- Inpatient notes exhibited higher redundancy (68.3%) than outpatient notes (60.7%).
- Redundancy varied by specialty, with Pediatrics (74.88%), Psychiatry (84.17%), and Specialty Internal Medicine (67.51%) showing higher percentages, while Radiology (45.86%) and Surgery (55.05%) had lower percentages.
Limitations and Future Work
- Single Institution Study: The research was conducted at a single institution, which might limit the generalizability of the findings to other healthcare systems.
- Annotation Challenges: Creating a high-quality gold standard for "new" information is inherently challenging and subjective, and the authors acknowledge the difficulties in annotation.
- Future Directions: The authors plan to develop more efficient classifiers for clinical relevance and explore how different roles (physicians, nurses) and specialties perceive the utility of new information visualization cues. They also suggest investigating differences in opinion regarding the utility of new information visualization cues for clinical notes.
Summary updated