TL;DR
This paper tackles the challenge of few-shot biomedical named entity recognition (BioNER), where labeled data is scarce. They propose a novel approach that reframes BioNER as a machine reading comprehension (MRC) task, leveraging carefully constructed "grape" demonstrations to significantly improve model performance compared to traditional sequence labeling methods.
What Problem Does This Paper Address?
Biomedical named entity recognition (BioNER) is crucial for extracting valuable information from vast amounts of medical literature. However, traditional deep learning models for BioNER require extensive manually labeled data, which is expensive and time-consuming to obtain, especially in specialized domains like biomedicine. Few-shot learning aims to address this by enabling models to perform well with very limited labeled examples. Existing few-shot approaches often rely on auxiliary supervision like dictionaries or rules, which can be noisy or computationally expensive. This paper seeks to improve few-shot BioNER by proposing a more effective demonstration-based learning strategy within the machine reading comprehension (MRC) framework.
Methods
The core of this paper's contribution lies in reframing BioNER as a Machine Reading Comprehension (MRC) problem. Instead of traditional token-level classification, they treat it as a task of extracting answer spans from a given context based on a query. This is achieved by transforming input sentences and their corresponding labels into a (Context, Query, Answer) triple format.
A key innovation is their "grape" demonstration construction method. This method prioritizes sentences with the highest entity density from the training data to serve as demonstrations. The intuition is that denser demonstrations provide richer contextual information for the model. They use a density formula based on the number of entities and sentence length to select these demonstrations.
For the actual model, they employ a BERT-based architecture (specifically PubmedBERTlarge) and train two binary classifiers to predict the start and end positions of entity spans. The loss function combines the cross-entropy losses for predicting start and end positions. During inference, they use a nearest-match principle to align predicted start and end spans, as the datasets used do not have overlapping entities.
Datasets
The proposed method is evaluated on six publicly available biomedical datasets:
- BC4CHEMD (chemicals)
- BC5CDR-Chemical (chemicals)
- BC5CDR-Disease (diseases)
- NCBI-Disease (diseases)
- BC2GM (genes and proteins)
- JNLPBA (biomedical entities)
These datasets were pre-processed and used to simulate few-shot scenarios by creating training sets with either 25 or 50 sentences per class, along with a development set of 100 sentences and the original test sets.
Key Insights and Ideas
- MRC is a powerful paradigm for few-shot BioNER: Reframing BioNER as an MRC task, rather than traditional sequence labeling, significantly boosts performance in low-resource settings. This is attributed to the MRC framework's ability to leverage contextual understanding more effectively.
- "Grape" demonstrations are more effective than "popular" ones: Selecting demonstrations based on entity density ("grape" demonstrations) generally leads to better performance than using demonstrations based on the most frequent entity ("popular" demonstrations), especially for chemical and protein entities.
- Demonstrations are crucial: The presence of well-constructed demonstrations, even in limited quantities, dramatically improves few-shot BioNER performance compared to models trained without them.
- Batch size matters significantly in extreme few-shot settings: For very limited data (e.g., 25-shot), a batch size of 1 is found to be optimal, suggesting that smaller batches allow models to better utilize the scarce training samples.
- Few-shot learning can rival fully supervised methods: The proposed MRC-based approach demonstrates competitive performance even when compared to fully supervised models that utilize vastly more annotated data.
Key Results
- In 25-shot learning, the proposed BERT-MRC grape method achieved average F1 scores of 61.7%, 84.1%, 69.1%, 70.1%, 50.6%, and 59.9% on the six benchmark datasets, showing improvements over baseline methods.
- In 50-shot learning, the average F1 scores further improved to 73.1%, 86.8%, 76.1%, 75.6%, 61.7%, and 65.4%.
- The BERT-MRC grape model consistently outperformed BERT-CRF models (both vanilla and demonstration-based) across most datasets and shot settings, highlighting the advantage of the MRC framework.
- When compared to GPT-4 (using prompts), the fine-tuned BERT-MRC grape model achieved higher F1 scores on several datasets in both 25-shot and 50-shot scenarios.
Limitations and Future Work
- Dataset-specific performance: While grape demonstrations generally perform well, "popular" demonstrations were found to be more effective for disease entities in the 25-shot scenario. This suggests that the optimal demonstration strategy might be dataset-dependent.
- Batch size sensitivity: The significant impact of batch size, particularly the advantage of a batch size of 1 in extreme few-shot settings, indicates that careful hyperparameter tuning is crucial.
- Computational cost of GPT-4: The paper notes that while GPT-4 is powerful, its extensive parameters and usage conditions make BERT-MRC grape a more practical and applicable solution in many scenarios, especially with limited annotated data.
- Future work: The authors suggest that further exploration into more sophisticated demonstration selection strategies and the application of this MRC-based approach to other biomedical NLP tasks could be promising avenues. The potential for few-shot learning to significantly impact assisted healthcare is also highlighted.
Summary updated