Document-level Clinical Entity and Relation extraction via Knowledge Base-Guided Generation

Kriti Bhattarai, Inez Y. Oh, Zachary B. Abrams, Albert M. LaiACL 2024

TL;DR

This paper introduces a novel approach to extract clinical entities and their relationships from medical texts by integrating the Unified Medical Language System (UMLS) with large language models (LLMs) like GPT. By guiding LLMs with UMLS concepts, the authors significantly improve the accuracy of information extraction, outperforming standard methods like Retrieval Augmented Generation (RAG).

What Problem Does This Paper Address?

Extracting precise and contextual information from clinical documents is crucial for healthcare applications, but current LLMs, despite their impressive contextual understanding, often struggle with the nuances and specialized terminology of the medical domain. This can lead to errors in identifying medical concepts, especially those that are less common or represented by abbreviations and brand names. Existing knowledge bases like UMLS contain a wealth of structured medical knowledge, but effectively integrating this knowledge with the generative capabilities of LLMs for tasks like entity and relation extraction remains an open research challenge.

Methods

The core of this work is a framework that leverages the strengths of both LLMs and knowledge bases. The authors first use MetaMap to identify UMLS concepts within a given clinical text. These identified concepts are then filtered to focus on medication-related terms. This filtered list of UMLS concepts is then incorporated into the prompts provided to LLMs (specifically GPT-4-32k and GPT-3.5-turbo) in a few-shot learning setting. This "knowledge base-guided generation" aims to steer the LLM's attention and improve its ability to recognize and extract specific clinical entities and their relationships. They also compare this approach against a standard Retrieval Augmented Generation (RAG) method.

Datasets

The study evaluates its framework on two publicly available datasets:

  • n2c2 Dataset: A curated dataset from the National NLP Clinical Challenges, consisting of 303 deidentified discharge summaries from the MIMIC-III critical care database. This dataset includes annotations for medication-related entities and their relationships.
  • ADE Dataset: This dataset comprises MEDLINE case reports annotated with information on medications, dosages, and adverse drug events, including their relationships.

Key Insights and Ideas

  • UMLS as a "Medical Lexicon" for LLMs: The paper demonstrates that LLMs, while good at general context, benefit immensely from explicit guidance on domain-specific terminology. UMLS acts as a crucial "lexicon" that helps LLMs recognize entities they might otherwise miss.
  • Prompt Engineering with Knowledge: Simply providing raw text to an LLM is less effective than crafting prompts that are augmented with relevant, structured knowledge. The paper shows that integrating UMLS concepts directly into the prompt leads to better performance than standard RAG.
  • Targeted Concept Filtering is Key: Not all UMLS concepts are equally relevant for every task. The authors' strategy of filtering UMLS concepts to focus on specific categories (like "pharmacologic substance") is crucial for effective augmentation.
  • GPT-4's Superiority: The results consistently show that the more powerful GPT-4 model benefits more significantly from UMLS integration, achieving higher performance gains.

Key Results

The proposed UMLS-guided generation approach consistently outperformed baseline LLMs and the RAG method across both datasets.

  • n2c2 Dataset: GPT-4-32k with UMLS integration showed a 4% improvement in F1-score compared to GPT-4-32k without UMLS.
  • ADE Dataset: GPT-4-32k with UMLS integration achieved a significant 12% improvement in F1-score compared to GPT-4-32k without UMLS.
  • Across all entity-entity relation pairs, performance improvements were observed for both GPT models when augmented with UMLS.
  • The RAG approach, while showing some improvement with UMLS, did not reach the performance levels of the direct UMLS-guided generation with GPT-4.

Limitations and Future Work

The authors acknowledge several limitations:

  • Focus on Medications: The current framework is primarily focused on medication-related entities. Future work could extend this to other UMLS entity types and explore richer semantic metadata within UMLS.
  • Limited Model Comparison: The study was limited to GPT-4 and GPT-3.5-turbo. Future research could include a broader range of LLMs, including domain-specific models like BioGPT.
  • RAG Limitations: The RAG approach's performance was potentially limited by how UMLS data is chunked for indexing, which might exclude relevant concepts. Further investigation into RAG strategies is suggested.

The authors envision future work exploring additional LLMs and deeper integration of UMLS's semantic capabilities to extract a wider array of clinical entities and relationships, ultimately enhancing LLM utility in biomedical text analysis.

Summary updated

BibTeX
@inproceedings{bhattarai2024document,
  author = {Kriti Bhattarai and Inez Y. Oh and Zachary B. Abrams and Albert M. Lai},
  title = {{Document-level Clinical Entity and Relation extraction via Knowledge Base-Guided Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure