TL;DR
This paper investigates how to best integrate external biomedical knowledge into Large Language Models (LLMs) for relation extraction tasks. They found that how knowledge is presented (e.g., closer to the task instance) matters more than its inherent quality, and this approach can help smaller LLMs perform comparably to much larger ones.
What Problem Does This Paper Address?
Large Language Models (LLMs) show great promise in various domains, including medicine. However, deploying them locally raises privacy concerns, and retraining them on specialized medical data is costly. External knowledge bases (KBs) offer a way to enhance LLMs without extensive retraining or exposing sensitive data. While integrating KBs into LLM prompts is common, the methods for presenting this knowledge and their impact on performance, especially in specialized domains like biomedicine, remain underexplored. This paper aims to fill this gap by systematically analyzing different knowledge presentation strategies and their effectiveness.
Methods
The core of this research lies in exploring different ways to inject knowledge from the Unified Medical Language System (UMLS) into LLM prompts for biomedical relation extraction.
- Knowledge Extraction: They use QuickUMLS to map entities from a given text to their Concept Unique Identifiers (CUIs) in UMLS. From these CUIs, they extract direct and one-hop relationships.
- Knowledge Verbalization: These extracted triples are then converted into natural language "knowledge statements" to be more easily understood by LLMs.
- Knowledge Injection Strategies: The paper proposes and evaluates three main methods for injecting these knowledge statements into prompts:
- Bag of KI: All knowledge statements are prepended to the entire prompt as a general background.
- Instance-based KI: Knowledge statements are prepended directly before each individual task instance (demonstration or test example).
- Refined KI: This method builds on Instance-based KI but further prunes knowledge statements based on semantic similarity to the instance, using a trained model (PUBMEDBERT) and a similarity threshold.
- Model: They primarily use MISTRAL (a 7B parameter LLM) within an In-Context Learning (ICL) framework. They also use BIOMISTRAL (a MISTRAL model pre-trained on biomedical text) for comparison.
- Evaluation: Performance is measured using Macro F₁ scores.
Datasets
The study uses four biomedical relation extraction datasets:
- ChemProt: An English dataset.
- DDI: An English dataset for drug-drug interactions.
- ADE-de: A subset of ADE in German.
- ADE-fr: A subset of ADE in French.
Key Insights and Ideas
- Position Matters More Than Quality: The most significant finding is that the position of the knowledge statements within the prompt is more crucial for performance improvement than the inherent quality or relevance of the knowledge itself. Placing knowledge closer to the specific task instance (Instance-based KI) yields substantial gains.
- Bridging the Gap for Smaller LLMs: This knowledge injection approach allows smaller LLMs (like the 7B MISTRAL) to achieve performance comparable to much larger models (like GPT-3.5) on certain tasks, making powerful NLP more accessible.
- Beyond Relevance: Even irrelevant knowledge statements, when positioned correctly, can improve performance, suggesting that the addition of tokens and the change in prompt structure play a significant role.
- Prompt Formatting is Key: Different entity marking strategies (ordered, entity-type, decorated) have varying effectiveness across datasets, highlighting the nuanced nature of prompt engineering.
Key Results
- Instance-based KI Outperforms Bag of KI: Instance-based KI consistently shows better performance than Bag of KI across all datasets, with improvements ranging from a few percentage points to over 20% F₁ on ChemProt.
- Smaller LLMs Catch Up: MISTRAL (7B) with Instance-based KI performs significantly better than its base setup (ICL w/o KI) and gets closer to larger models like GPT-3.5-TURBO on ChemProt and GPT-3.5 on DDI.
- Refined KI vs. Instance-based KI: The performance gains from Refined KI over Instance-based KI are marginal, suggesting that the strict pruning of "lower quality" knowledge might not be as beneficial as initially hypothesized.
- Positioning is Crucial: Even irrelevant knowledge sources (UMLS instance-unrelated, corpus-unrelated, Bible) show performance improvements when placed close to instances, indicating the impact of prompt structure and token count.
Limitations and Future Work
- Hyperparameter Tuning: Hyperparameters were tuned in a cascaded manner, which might not be optimal.
- Entity Linking Bottleneck: Entity linking can be a bottleneck, especially with noisy or informal user-generated data.
- Prompt Length: The effect of prompt length and the impact of inserting irrelevant tokens require further investigation.
- Multilingual Prompts: While the study used mixed-language prompts (English instructions with French/German samples), the full impact of this multilingual aspect on performance needs more exploration.
- Future Work: The authors suggest further research into the effect of irrelevant tokens and their placement, as well as domain-specific studies on prompt length.
Summary updated