TL;DR
This paper introduces CLEAR, a novel retrieval-augmented generation (RAG) pipeline that leverages clinical entities for more efficient and accurate information extraction from electronic health records (EHRs). CLEAR significantly reduces token usage and inference time compared to existing RAG and full-note approaches, while achieving comparable or improved performance.
What Problem Does This Paper Address?
Electronic health records (EHRs) contain a wealth of unstructured clinical notes that hold valuable information beyond structured fields. Extracting this information is crucial for research, quality improvement, and clinical decision-making. Traditional methods like rule-based systems and even modern natural language processing (NLP) techniques, including large language models (LLMs) with retrieval-augmented generation (RAG), face challenges. LLMs, while powerful, struggle with the long context windows of clinical notes, leading to information loss or inefficient processing. Standard RAG methods often rely on embedding entire text chunks, which can be suboptimal as it might retrieve irrelevant information or miss crucial context. This inefficiency translates to higher computational costs and slower processing times, hindering widespread adoption in clinical settings.
Methods
The core innovation is the Clinical Entity Augmented Retrieval (CLEAR) pipeline. Unlike traditional RAG that embeds text chunks, CLEAR focuses on retrieving information based on clinical entities. The pipeline proceeds as follows:
- Entity Recognition (NER): A zero-shot NER model (Flan-T5-XXL) is used to identify clinical entities within the input clinical notes.
- Entity Selection: Identified entities are filtered to retain those relevant to a specific "target entity" (e.g., "food insecurity"). This filtering is done using Bio+Clinical BERT embeddings and cosine similarity, followed by an LLM-based refinement.
- Entity Augmentation: To mitigate missed entities from NER, ontologies (UMLS) and LLMs (GPT-4) are used to augment the entity list with synonyms and related concepts.
- Information Retrieval: The augmented entity list is used to retrieve relevant note chunks. This is achieved using a "Target Matcher" that finds note segments containing these entities, with a focus on a context window around them.
- Downstream Task: The retrieved note chunks are then passed to an LLM for the final information extraction task (e.g., determining if a patient experiences food insecurity).
This entity-centric approach aims to retrieve more targeted and relevant information, thereby improving efficiency.
Datasets
The study utilized two real-world EHR-derived datasets:
- Stanford MOUD Dataset: This dataset comprises clinical notes from patients treated for opioid use disorder at Stanford Hospital. It was used for training and evaluating the extraction of 13 clinical variables, including substance use disorders, mental health conditions, social determinants of health, and chest radiograph findings.
- CheXpert Dataset: This dataset consists of radiology reports from Stanford Hospital, with programmatic labels for five clinical entities related to chest X-rays (e.g., cardiomegaly, pulmonary edema).
The authors also mention using the NCBI Disease Dataset for evaluating the zero-shot NER performance.
Key Insights and Ideas
- Entity-centric retrieval is more efficient: Focusing retrieval on specific clinical entities, rather than generic text chunks, leads to significantly more targeted and relevant information retrieval. This is the core hypothesis and finding of the paper.
- LLMs can be "distilled" for clinical tasks: The outputs from CLEAR can be used to weakly label data, enabling the fine-tuning of smaller, more efficient BERT-sized models for clinical information extraction, achieving performance comparable to larger LLMs.
- NER is crucial for effective RAG: Removing the initial NER step from CLEAR significantly degrades downstream information extraction performance, highlighting the importance of entity identification for guiding retrieval.
- Augmentation is key to robustness: Using ontologies and LLMs to augment entities missed by NER is vital for ensuring comprehensive information retrieval, as false negatives can lead to complete information loss.
Key Results
CLEAR demonstrated substantial improvements across several metrics:
- Performance: Achieved average F1 scores of 0.90 for CLEAR, compared to 0.86 for embedding RAG and 0.79 for full-note approaches across 18 variables and six LLMs.
- Efficiency:
- Token Usage: Reduced token usage by over 70% compared to full-note and embedding RAG approaches.
- Inference Time: Achieved over 70% reduction in inference time. Average inference times were 4.95s (CLEAR) vs. 17.41s (embedding RAG) vs. 20.08s (full-note).
- Model Queries: Reduced average model queries per note from 4.94 (embedding RAG) and 4.18 (full-note) to 1.68 for CLEAR.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Task Specificity: The current evaluation is limited to clinical variable extraction. Future work should explore CLEAR's performance on other tasks like summarization, question answering, and clinical reasoning, potentially using benchmarks like MedAlign.
- Chunk Size Optimization: While CLEAR outperformed chunk embedding even with adjusted chunk sizes, further research into optimal chunk size tuning for both CLEAR and chunk embedding methods is warranted.
- Multi-Note Processing: The current CLEAR pipeline processes one note chunk at a time. Adapting it to combine chunks from multiple notes for LLM inference could be beneficial for tasks requiring cross-note information.
- Prompt Tuning: Further optimization of prompts for NER, entity selection, and LLM augmentation could enhance CLEAR's performance.
- Data Drift: The study noted potential data drift due to the COVID-19 pandemic, which might have influenced the prevalence of certain conditions in the testing set. Future work should consider temporal aspects of data.
- Model Quantization: Incorporating model quantization techniques could further improve efficiency and scalability without significantly compromising performance.
Summary updated