TL;DR
This paper tackles the challenge of automating clinical risk score calculation, which is currently time-consuming and relies on manual data extraction. The authors propose a novel approach using off-the-shelf Question Answering (QA) models and biomedical ontology linkers, demonstrating that this method can achieve comparable or even better performance than traditional heuristic-based approaches, paving the way for more scalable and generalizable risk score automation.
What Problem Does This Paper Address?
Clinical risk scores are essential tools for stratifying patients and informing medical decisions. However, calculating these scores often requires significant manual effort to extract data from electronic health records (EHRs). Previous attempts to automate this process have largely focused on specific risk scores and relied on manually curated rules or heuristics, which are rigid and difficult to adapt to new scores or variations. This paper aims to develop more generalizable methods for automating risk score calculation by leveraging modern NLP techniques, reducing the need for expert-defined rules.
Methods
The authors propose a two-pronged approach:
- Structured Data Information Retrieval: For structured EHR data, they use MedCAT (a biomedical ontology linker) to identify relevant concepts (CUIs) for each risk score element. These CUIs are then mapped to institution-specific billing codes (ICD9 for MIMIC-III). SQL queries are then constructed to retrieve Yes/No answers based on the presence or absence of these codes in the patient's record.
- Unstructured QA Model: For unstructured EHR data (clinical notes), they employ ClinicalBERT, a transformer-based model pre-trained on MIMIC-III. This model is fine-tuned on the emrQA dataset. To improve performance, they implement negative sampling for unanswerable questions and augment training data with parent MeSH hierarchy entities for vague risk score elements. The model predicts relevant text spans, and MedCAT's negation detector is used to determine the final answer.
Crucially, they also explore a combined approach where structured data predictions are prioritized, and unstructured data predictions are used as a fallback.
Datasets
- MIMIC-III: The primary dataset used for annotation and evaluation. The authors randomly sampled 100 patients from MIMIC-III.
- emrQA: Used for fine-tuning the ClinicalBERT QA model.
A new dataset was created by annotating 100 MIMIC-III patients with elements of the CHA2DS2-VASc and PERC scores. This dataset is intended to be released to the community.
Key Insights and Ideas
- QA as a Generalizable Solution: The study highlights the potential of QA models to automate risk score elements, particularly for more general categories like "surgery/trauma," where heuristic-based methods struggle. This suggests a shift towards more flexible and adaptable automation.
- Structured Data Strengths and Weaknesses: Chronic conditions are well-represented in structured data, leading to high performance for related risk score elements. However, acute events and symptoms without formal diagnoses are often missed, underscoring the need for unstructured data analysis.
- Limitations of Heuristics: The paper implicitly reinforces the limitations of heuristic-based approaches, which are rigid and require extensive manual effort to define and maintain.
- The Need for Clinical Knowledge: The QA model, while promising, still struggles with distinguishing hypothetical mentions from true affirmations and understanding nuanced medical terminology, pointing to the ongoing challenge of imbuing NLP models with deep clinical reasoning.
Key Results
- The QA-based approach on unstructured data alone showed improvements over a baseline NER-based approach (Bean et al., 2019) for several risk score elements.
- The combined structured and unstructured approach significantly outperformed the unstructured-only QA model and the baseline for many risk score elements, demonstrating the synergy between data modalities.
- For example, in the combined model, the F1 score for CHA2DS2-VASC CHF was 0.857, and for Hypertension, it was 0.923.
- The structured data retrieval component achieved very high precision and recall for certain elements like CHA2DS2-VASC CHF (R=1.0, P=0.94, F1=0.97) and Hypertension (R=0.97, P=0.81, F1=0.89).
- However, performance was notably lower for elements like "Stroke/TIA/TE" and "PE/DVT," where the QA model struggled with negation detection and understanding context.
Limitations and Future Work
- Negation and Hypothetical Detection: The authors acknowledge that the QA model struggles with accurately detecting negations and distinguishing hypothetical mentions from true affirmations, leading to false positives. This is a key area for future improvement.
- Clinical Knowledge Integration: The model's difficulty in distinguishing between broad and specific definitions of conditions (e.g., vascular disease) and its inability to use contextual clues effectively highlight the need for better clinical knowledge integration.
- Generalizability to Other Risk Scores: While the approach is designed for generalizability, further validation on a wider range of risk scores is needed.
- Dataset Release: The authors plan to release their annotated dataset to encourage further research in this area.
- Improved Negation/Hypothetical Detection and Clinical Knowledge Embeddings: These are explicitly mentioned as areas for future work.
Summary updated