TL;DR
This paper investigates how to best construct biomedical knowledge graphs from text using Large Language Models (LLMs). They found that smaller, fine-tuned encoder-decoder models outperform larger decoder-only models using in-context learning, and that a simple heuristic for controlling output hallucinations significantly boosts performance.
What Problem Does This Paper Address?
Extracting structured knowledge from text into formats like knowledge graphs (KGs) is crucial for AI systems to reason and answer questions. While Large Language Models (LLMs) show promise in this area, there's a lack of clear guidance for practitioners on how to best leverage them for biomedical text-to-graph tasks, especially under realistic computational constraints. Previous work often focused on comparing specific model architectures on general domains, leaving a gap in understanding how LLM characteristics (architecture, size), learning methods (fine-tuning vs. in-context learning), and pre-training data impact performance in the specialized biomedical domain.
Methods
The researchers explored two main approaches for adapting LLMs to the text-to-graph task:
- Fine-tuning: This involves further training a pre-trained model on a specific dataset to update its parameters. They applied this to smaller encoder-decoder models (T5 and BART).
- In-Context Learning (iCL): This method uses a pre-trained model without further training, instead providing a few examples within the prompt to guide its output. This was used for larger decoder-only models (Mistral and Llama-2).
A key methodological contribution is a simple truncation-based heuristic to control for "hallucinations" in the generated graphs. This heuristic automatically identifies the end of a generated knowledge graph by looking for specific tokens, avoiding the need for complex prompt engineering or tuning.
Datasets
The primary dataset used for evaluation was BioEvent, a benchmark comprising 10 popular biomedical datasets. The authors also mention using a relation extraction dataset called REBEL for additional pre-training of some models.
Key Insights and Ideas
- Biomedical KGs are Tough: The biomedical domain presents a unique challenge for text-to-graph extraction, potentially requiring more extensive examples for accurate knowledge extrapolation compared to general domains.
- Fine-tuning Wins for Smaller Models: Smaller encoder-decoder models that are fine-tuned demonstrate robust and strong performance, outperforming larger decoder-only models using iCL.
- Heuristic for Hallucination Control is Effective: A simple post-hoc heuristic to truncate model output significantly boosts performance for iCL models, offering a practical solution without expensive prompt engineering.
- Pre-training Matters Selectively: Only pre-training on a relation extraction dataset (REBEL) showed a consistent performance boost. Other pre-training data, including biomedical-specific data (Meditron), did not yield significant improvements for the tested models and methods.
- Scaling Laws Apply Differently: Power-law scaling was observed for model size in fine-tuned encoder-decoder models, while for decoder-only models with iCL, scaling was observed in the number of in-context examples.
Key Results
- Best Performance: The best overall performance was achieved by fine-tuned encoder-decoder models, specifically the largest T5 model.
- Heuristic Impact: The hallucination-control heuristic led to performance jumps of over 20 points across all Rouge scores for iCL models.
- Model Size Correlation: A clear positive correlation between model size and performance was observed for both T5 and Llama-2 families.
- Pre-training Data: Adopting the REBEL dataset for pre-training BART models resulted in universally superior results compared to other pre-training additions.
Limitations and Future Work
The authors acknowledge that their experimental setup was limited by computational resources, specifically a single RTX 8000 GPU. This prevented them from fine-tuning larger decoder-only models or mid-sized Mistral/Llama-2 families, which could potentially yield even better results. They also note that while their heuristic is effective, it might not be universally generalizable across all subsets of the task. Future work could explore prompt engineering and tuning for iCL models, investigate the impact of larger computational resources, and further analyze the specific challenges of the biomedical domain for KG extraction.
Summary updated