REAL: A Retrieval-Augmented Entity Linking Approach for Biomedical Concept Recognition

Darya Shlyk, Tudor Groza, Marco Mesiti, Stefano Montanelli, Emanuele CavalleriACL 2024

TL;DR

This paper introduces REAL, a novel approach for automatically identifying and linking biomedical concepts in scientific text to established ontologies. By combining Large Language Models (LLMs) with a retrieval-augmented generation (RAG) strategy, REAL overcomes limitations of traditional methods, achieving competitive performance in concept recognition without requiring domain-specific training.

What Problem Does This Paper Address?

Accurately identifying and linking biomedical concepts (like diseases, genes, or phenotypes) within unstructured text is crucial for many downstream applications, such as improving genetic disease diagnostics. Traditional methods often rely on dictionary lookups or require extensive, task-specific training data, which is scarce in the biomedical domain. While general-purpose LLMs show promise with their zero-shot capabilities, they can suffer from outdated knowledge and factual inaccuracies. This paper tackles the challenge of improving the accuracy and reliability of biomedical concept recognition by leveraging the strengths of LLMs while mitigating their weaknesses through external knowledge integration.

Methods

The core of REAL is a Retrieval-Augmented Generation (RAG) approach applied to biomedical concept recognition (CR). CR itself is broken down into two subtasks: Named Entity Recognition (NER) and Entity Linking (EL).

  1. Ontology Indexing: First, the biomedical ontology (e.g., Human Phenotype Ontology - HPO) is indexed. Concepts are converted into vector embeddings and stored in a searchable vector database (ChromaDB). This indexing is a one-time pre-processing step.
  2. Zero-Shot NER: For a given text, an LLM is prompted to extract mentions of target biomedical entities (e.g., phenotypes) and generate a short definition for each mention. This is done without any domain-specific fine-tuning.
  3. Retrieval-Augmented Entity Linking (EL): For each extracted mention and its definition, a semantic similarity search is performed against the indexed ontology. This retrieves the top-k most similar concepts from the ontology. The LLM is then prompted to select the best matching concept from this retrieved set, effectively performing a multiple-choice selection. This RAG step injects up-to-date, relevant knowledge from the ontology into the LLM's decision-making process.

The approach is designed to be versatile and adaptable to different biomedical domains by simply changing the target entity types and the ontology used.

Datasets

The paper evaluates REAL on two publicly available benchmark datasets for clinical phenotyping and phenotype annotation:

  • HPO GSC+: A dataset of manually annotated PubMed abstracts covering human phenotype ontology (HPO) IDs.
  • BIOC-GS: A dataset of manually annotated clinical observations of phenotypes identified during dysmorphology physical examinations.

The Human Phenotype Ontology (HPO) itself serves as the reference resource for grounding the identified concepts.

Key Insights and Ideas

  • RAG for Biomedical CR is Promising: The study demonstrates that integrating external knowledge bases via RAG significantly enhances the performance of LLMs for biomedical concept recognition, addressing issues of factual accuracy and outdated knowledge.
  • Zero-Shot NER & EL Efficiency: The approach avoids the need for extensive domain-specific training data by leveraging zero-shot prompting for NER and a retrieval-augmented, few-shot learning approach for EL.
  • LLM-Assisted EL Improves Precision: Comparing two grounding strategies, using the LLM to select the best candidate concept (REAL-GPT3.5) yielded more precise results than simply picking the first retrieved concept (REAL-1st HIT), indicating the LLM's ability to reason and filter out spurious extractions.
  • Ontology Indexing is Key: The effectiveness of the retrieval step hinges on the quality of the ontology index and the ability to generate semantically rich embeddings for concepts.

Key Results

REAL-GPT3.5 achieved competitive performance on both the HPO GSC+ and BIOC GS datasets, demonstrating strong F1 scores at both document and mention levels. Specifically:

  • REAL-GPT3.5 achieved the best F1 scores at both mention and document levels across both datasets.
  • While SPIRES (a prompt-based alternative) achieved high precision, REAL-GPT3.5 significantly improved recall due to the retrieval mechanism.
  • REAL-1st HIT (using only the first retrieved concept for linking) showed comparable F1 scores to SPIRES, balancing precision and recall.
  • REAL-GPT3.5 showed improved precision over REAL-1st HIT, highlighting the benefit of LLM-based candidate selection.

Limitations and Future Work

  • LLM API Dependency: The current implementation relies on OpenAI's API, which hinders reproducibility. Future work aims to address this by using local installations of open-source LLMs.
  • Cross-Domain Generalizability: The current evaluation is limited to phenotype recognition. Future research will focus on a comprehensive cross-domain evaluation to assess the generalizability of REAL.
  • NER Prompt Sensitivity: The performance of the NER prompt is critical. Poor NER results propagate through the pipeline. While the current prompt shows good coverage, further refinement could be beneficial.
  • Annotation Idiosyncrasies: Differences in annotation styles across datasets can lead to false positives and negatives, suggesting a need for additional post-processing of NER results.
  • GPT-4 Potential: The experiments were limited to GPT-3.5, and it is anticipated that using GPT-4 could yield even better results.

Summary updated

BibTeX
@inproceedings{shlyk2024real,
  author = {Darya Shlyk and Tudor Groza and Marco Mesiti and Stefano Montanelli and Emanuele Cavalleri},
  title = {{REAL: A Retrieval-Augmented Entity Linking Approach for Biomedical Concept Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure