TL;DR
This paper introduces WeLT, a novel cost-sensitive learning approach to address class imbalance in biomedical named entity recognition (BioNER) fine-tuning. By re-weighting minority classes, WeLT significantly improves performance across various BioNER datasets and pre-trained models, outperforming existing methods.
What Problem Does This Paper Address?
Fine-tuning large pre-trained language models (PLMs) like BioBERT has become the standard for achieving state-of-the-art results in biomedical natural language processing (NLP) tasks, particularly Named Entity Recognition (NER). However, a significant challenge arises from the inherent class imbalance in real-world biomedical datasets. These datasets, often curated by domain experts, tend to have a vast majority of "background" tokens (labeled 'O') and a scarcity of tokens belonging to specific biomedical entities (labeled 'B' for beginning and 'I' for inside). This imbalance can lead BioNER models to become biased towards the majority class, resulting in poor performance and misclassification of crucial biomedical entities. While class imbalance is a known issue in NLP, its impact and effective mitigation strategies for biomedical NER remain underexplored.
Methods
The core contribution of this paper is WeLT (Weighted Loss Trainer), a cost-sensitive fine-tuning approach designed to tackle class imbalance in BioNER. WeLT operates by modifying the loss function during the fine-tuning stage. Instead of treating all misclassification errors equally, WeLT assigns higher weights to minority classes (B and I tags) and lower weights to the majority class (O tag).
The re-scaled class weights are calculated using a generic formula based on the inverse ratio of each class's distribution within the training dataset. Specifically, the weight for a class 'x' is computed as:
$W_x = \frac{1}{I_x}$
where $I_x$ is the number of instances for class 'x'. These initial weights are then normalized using a Softmax function to ensure they sum to 1, creating a weighted cross-entropy loss. This approach avoids the need for hyperparameter tuning for the weighting scheme itself, making it more adaptable. WeLT is integrated into the training pipeline by extending existing trainer classes and overriding the compute_loss function.
Datasets
The authors evaluated WeLT on eight biomedical gold-standard datasets for BioNER:
- NCBI Disease: Focuses on disease mentions.
- BC5CDR-Disease & BC5CDR-Chemical: Datasets for chemical and disease relation extraction.
- BC4CHEMD: For chemical and drug name recognition.
- BC2GM: For gene and protein mention recognition.
- Linnaeus: For species name annotation.
- BioRED-Disease & BioRED-Chemical: Datasets for biomedical relations, focusing on chemical and disease instances.
The paper also provides a table (Table 11) detailing the number of sentences in these datasets for training, validation, and testing.
Key Insights and Ideas
- Class imbalance is a critical factor in BioNER: The paper strongly demonstrates that ignoring class imbalance leads to suboptimal performance in biomedical NER tasks.
- WeLT offers a generic and effective solution: The proposed WeLT method provides a principled way to re-weight classes without requiring extensive hyperparameter tuning for the weighting scheme itself.
- Domain-specific pre-training is beneficial, but class imbalance still matters: Even with models pre-trained on biomedical data (like BioBERT, PubMedBERT), addressing class imbalance during fine-tuning yields significant improvements.
- The "O" tag is the dominant majority class: In the IOB tagging scheme, the 'O' tag (non-entity) vastly outnumbers the 'B' and 'I' tags, making it the primary target for re-weighting.
- Error analysis reveals specific challenges: The paper identifies three types of errors: entities predicted but not annotated, entities annotated but not predicted, and overlapping span errors, with abbreviated entities being a notable issue.
Key Results
WeLT consistently outperformed vanilla fine-tuning across all tested BioNER datasets and pre-trained models. When compared to other existing weighting schemes (Inverse of Number of Samples, Inverse of Square Root of Number of Samples, and Effective Number of Samples), WeLT achieved the best or second-best F1-scores in most experiments.
For instance, on the Linnaeus dataset, which is highly skewed, WeLT achieved the best scores for most models. On the NCBI dataset, the least imbalanced, WeLT also performed very well, often achieving the best scores. The paper presents extensive results in Tables 2-10, showing F1-scores often in the range of 80-90% for WeLT, with notable improvements over vanilla fine-tuning.
Limitations and Future Work
The authors acknowledge several limitations:
- Language Specificity: The current evaluation was limited to English datasets, and its applicability to other languages is yet to be explored.
- Dataset Size: The performance on significantly larger training datasets (beyond 30,683 examples, like BC4CHEMD) was not extensively evaluated.
- Downstream Task Specificity: WeLT was evaluated solely for BioNER. The authors hypothesize that it can be adapted to other tasks and domains with skewed datasets.
Future work suggested includes:
- Evaluating WeLT on larger datasets.
- Exploring its application to other NLP tasks and non-biomedical domains.
- Investigating the impact of dataset size as an additional factor in weighting schemes.
- Addressing specific error types identified in the analysis, such as abbreviated entities, through knowledge enrichment or active learning.
Summary updated