TL;DR
This paper introduces a novel approach to automatically identify key risk factor domains within psychiatric electronic health records (EHRs) using natural language processing (NLP). By developing a robust topic extraction pipeline that incorporates multi-word expressions (MWEs), the authors aim to build more accurate and interpretable readmission risk classifiers for psychosis patients, a critical step towards improving patient care and reducing healthcare costs.
What Problem Does This Paper Address?
Readmission after psychiatric hospitalization is a significant problem, leading to disruption for patients and increased healthcare costs. Predicting which patients are at high risk of readmission is crucial for targeted interventions, but it's challenging due to the complex and varied language used in psychiatric EHRs. Traditional methods like manual chart review are time-consuming and don't scale well. While NLP has shown promise in other medical fields, its application in psychiatry is hindered by unique challenges: highly context-dependent vocabulary (e.g., "feeling really great" could indicate mania), the prevalence of multi-word expressions (MWEs) that lose meaning when broken down, and the varied narrative structures used to describe the same phenomena. Furthermore, clinical narratives can contain information about future treatment plans that might appear related to risk factors but aren't indicative of the patient's current state.
Methods
The core of this work is a data pipeline for topic extraction from psychiatric EHRs. They created a system that first uses a lexicon of keywords and MWEs identified by domain experts to categorize paragraphs into seven specific risk factor domains: Appearance, Mood, Interpersonal, Occupation, Thought Content, Thought Process, and Substance.
Key methodological contributions include:
- Multi-Word Expression (MWE) Integration: Recognizing that many clinical concepts are expressed as MWEs (e.g., "short attention span"), they incorporated these into their analysis. By treating MWEs as single units, they amplified their predictive value.
- TF-IDF Vectorization and Dimensionality Reduction: They used TF-IDF to represent text and then applied Singular Value Decomposition (SVD) to reduce the vector space to 100 dimensions, a common practice to improve classifier performance.
- Neural Network Classifiers: They trained both Multi-Layer Perceptron (MLP) and Radial Basis Function (RBF) neural networks to classify paragraphs into the identified risk factor domains.
- Domain-Specific Thresholding: To handle the varying homogeneity of language across domains, they developed a formula to calculate threshold similarity scores for each domain, allowing for more flexible and adaptive classification. Paragraphs not meeting a threshold are classified as "Other."
Datasets
- Target Data: A corpus of approximately 240,000 EHR paragraphs from 220 patients in the OnTrackTM program at McLean Hospital. This program focuses on early intervention for adults aged 18-30 experiencing their first episodes of psychosis.
- Training Data: A larger corpus of approximately 100,000 paragraphs (7,000,000 tokens) derived from the Research Patient Data Registry (RPDR), a centralized data repository from Partners HealthCare. This dataset was used to train the vector space model and was categorized using a clinician-developed lexicon.
- Annotation Data: A subset of 1,654 paragraphs from the target Meditech data, manually annotated by three domain-expert clinicians to serve as the gold standard for evaluating their models.
Key Insights and Ideas
- MWEs are Crucial: The inclusion of MWEs significantly boosted classification performance (by 15%), highlighting their importance in capturing clinical meaning in psychiatric text.
- Domain Homogeneity Varies: Different risk factor domains exhibit distinct levels of linguistic homogeneity. Some domains (like Substance) are more clearly defined by specific language, while others (like Interpersonal and Mood) are more diffuse and prone to overlap.
- MLP vs. RBF Performance: While RBF networks are often favored for similar tasks, the authors found that an MLP network performed marginally better with less preprocessing, particularly in domains with significant overlap.
- Context is Key (and Often Missing): The study highlights the difficulty in classifying paragraphs when they are decontextualized. This lack of surrounding information can lead to ambiguity, especially for terms that can apply to multiple domains (e.g., "blunted/restricted").
Key Results
- Overall Performance: The best-performing models (MLP and RBF with MWEs) achieved an F1 score of 0.681 and 0.645, respectively, on the overall classification task.
- Domain-Specific Performance: The models achieved the highest per-domain F1 scores for Substance (around 0.79-0.80) and the lowest for Interpersonal and Mood (around 0.45-0.50).
- Inter-Annotator Agreement: The agreement among human annotators was "generally good" (Fleiss's Kappa of 0.575), indicating that the defined risk factor domains are reasonably well-understood and applicable.
- MLP Advantage: The MLP model with MWEs outperformed the RBF model with MWEs in several overlapping domains, suggesting its robustness in handling nuanced language.
Limitations and Future Work
- Limited Training Data Homogeneity: The training corpus, while large, might not fully capture the linguistic diversity of the target patient population, potentially leading to performance variations.
- Decontextualized Paragraphs: The annotation process removed paragraphs from their original context, which made it challenging for annotators to definitively assign domains in cases of ambiguity.
- Future Directions:
- Sequence Modeling: Incorporating sequence modeling to consider more contextual information within EHRs.
- Sentiment Analysis: Developing lexicon-based and corpus-based sentiment analysis tools to distinguish between clinically positive and negative phenomena within risk factor domains, enabling the tracking of patient improvement or deterioration.
- Integrating Structured Data: Factoring in additional structured data such as EEG biomarkers, MRI scans, personality assessments, and symptom scales to enhance the predictive power of the classifier.
Summary updated