TL;DR
This paper investigates whether state-of-the-art BERT models, even those pre-trained on medical data like BioBERT, possess sufficient clinical domain knowledge to perform complex reasoning tasks. The authors explore various methods to augment BERT with external medical knowledge but find that current techniques fail to significantly improve performance on clinical language inference, suggesting that unsupervised knowledge retrieval for complex reasoning remains a significant challenge.
What Problem Does This Paper Address?
Large language models (LLMs) like BERT have achieved remarkable success across many NLP tasks. However, their ability to perform nuanced reasoning, especially in specialized domains like medicine, is still an open question. While models like BioBERT are pre-trained on medical corpora, an analysis of their error patterns reveals a significant portion (40%) stems from a lack of domain-specific knowledge. This gap hinders their ability to accurately perform tasks like clinical language inference, which requires understanding complex relationships between medical concepts, treatments, and patient conditions. The paper aims to bridge this gap by exploring whether augmenting BERT with external medical knowledge can improve its performance on clinical inference tasks.
Methods
The authors investigate several approaches to integrate external medical knowledge into BERT models for clinical language inference:
- Implicit Knowledge Integration: This involves further pre-training existing BERT models (specifically BioBERT) on additional medical corpora. They experiment with two corpora: a medical-only subset of Wikipedia (Wikimed) and a popular medical textbook (Medbook). The goal is to see if this continued training implicitly enhances the model's understanding of fundamental medical knowledge.
- Explicit Knowledge Integration: This approach involves retrieving relevant text snippets from external medical corpora and appending them to the input sequence before feeding it to the BERT model for classification. They explore three methods for retrieval:
- Lexical Retrieval: Using the BM25 algorithm to find sentences that contain lemmas of infrequent medical entities from the premise and hypothesis.
- Lexical and Syntactic Retrieval: An extension of lexical retrieval that also considers dependency paths between premise and hypothesis lemmas to ensure a semantic relation.
- Joint Retrieval and Classification (End-to-End): Training a retriever module jointly with the classifier. This retriever learns to dynamically select the most informative text blocks from Wikipedia to aid classification. They reuse a pre-trained retriever from an inverse cloze task setup.
Datasets
- MedNLI: The primary dataset used for evaluating clinical language inference. It consists of premise-hypothesis pairs derived from patient notes in the MIMIC-III dataset, with hypotheses written by medical experts. The task is to classify the relationship between the premise and hypothesis as entailment, contradiction, or neutral.
- External Corpora:
- Wikimed: A medical-only subset of Wikipedia, containing approximately 40 million tokens.
- Medbook: A corpus derived from a popular medical textbook, containing approximately 3.6 million tokens.
- Wikipedia: The full Wikipedia corpus was used for pre-training the end-to-end retriever.
Key Insights and Ideas
- Domain Knowledge is Crucial for Clinical Inference: The analysis of BioBERT errors clearly demonstrates that insufficient domain knowledge is a primary driver of incorrect predictions in clinical language inference.
- Current Retrieval Methods Fall Short for Complex Reasoning: Despite the success of knowledge integration methods in tasks like open-domain question answering, they struggle to provide the necessary context for complex medical reasoning in an unsupervised setting.
- Implicit vs. Explicit Integration: Neither implicit knowledge integration (further pre-training) nor explicit knowledge integration (retrieval and appending) yielded significant improvements across the board.
- Wikipedia as a Knowledge Source: The study suggests that Wikipedia, even its full version, might be a more effective source of knowledge for this task than a specialized medical textbook, potentially due to its larger size and broader coverage.
- Unsupervised Retrieval is Hard: The core challenge lies in the unsupervised nature of the retrieval. Finding the exact piece of information that bridges the gap between a premise and hypothesis for inference, without explicit supervision, is proving to be a very difficult problem.
Key Results
- No Significant Performance Gains: Across all explored methods (implicit and explicit knowledge integration), the authors found no significant performance difference on the MedNLI test set compared to the baseline BioBERT model.
- Marginal Improvements: Some methods showed marginal improvements on the test set, but these were not consistently observed on the development set, suggesting they might be random effects.
- Error Analysis: The study identified "Insufficient domain knowledge" as the largest error category (40%) in the BioBERT model's performance on MedNLI.
- Retriever Limitations: Qualitative analysis of retrieved text blocks showed that even when sentences were related to the topic, they often failed to bridge the knowledge gap required for correct inference. The end-to-end retriever sometimes returned text blocks that were thematically relevant but not specifically helpful for the given instance.
Limitations and Future Work
- Limitations:
- The study was conducted in a "limited computation setting," which may have constrained the effectiveness of the methods.
- The unsupervised nature of the retrieval task makes it inherently challenging to find the precise supporting information.
- The end-to-end retriever was trained on the full Wikipedia, not just the medical subset, which might have introduced noise.
- Copyright limitations prevented scaling up the Medbook corpus.
- Future Work:
- Investigating methods that require more computational power to train models for longer durations to achieve convergence.
- Developing large, annotated corpora specifically for retrieving missing facts to improve the performance of retrieval algorithms.
- Exploring advanced evidence retrieval and knowledge integration setups to enhance knowledge-based reasoning in specialized domains.
Summary updated