BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition

Vera Pavlova, Mohammed MakhloufACL 2023

TL;DR

This paper introduces BIOptimus, a novel biomedical language model pre-trained using a curriculum learning approach and a new contextualized weight distillation technique. BIOptimus achieves state-of-the-art results on several biomedical Named Entity Recognition (NER) tasks while significantly reducing pre-training time.

What Problem Does This Paper Address?

Pre-trained language models (LMs) have revolutionized Natural Language Processing (NLP), but adapting them to specialized domains like biomedicine presents challenges. Biomedical text contains unique terminology and requires expert knowledge for annotation, making it difficult to create large, high-quality labeled datasets. While domain-specific LMs exist, there's a lack of comprehensive comparisons of pre-training methods to determine the most effective strategies for the biomedical domain. This paper aims to bridge this gap by investigating different pre-training approaches and proposing an optimized method for biomedical NER.

Methods

The authors explore several pre-training strategies for biomedical LMs:

  • Pre-training from scratch: Training a new model and vocabulary entirely on biomedical data.
  • Continued pre-training: Taking a general-domain LM (like BERT) and continuing its training on biomedical data.
  • Contextualized Weight Distillation: A novel technique to initialize weights for new biomedical tokens. Instead of random initialization, it distills existing weights from a general-domain model (BERT) within the context where the new tokens appear. This helps speed up pre-training and improves performance.
  • Curriculum Learning (CL): Inspired by the "easy-to-hard" learning principle, the authors propose a CL strategy for pre-training. This involves gradually increasing the complexity of the masked language modeling (MLM) task, starting with simpler token prediction and progressing to whole-word masking and finally removing corruption strategies. This approach aims to guide the model's learning more smoothly.

They also investigate the impact of masking rate, corruption strategy, and masking strategies (e.g., whole-word masking) on model performance.

Datasets

The primary dataset used for pre-training is the PubMed Corpus. For fine-tuning and evaluation, the authors use five NER datasets from the BLURB benchmark:

  • BC5-chem
  • BC5-disease
  • NCBI-disease
  • BC2GM
  • JNLPBA

Key Insights and Ideas

  • Contextualized Weight Distillation is Key: Initializing new biomedical vocabulary weights by distilling from existing contextual embeddings significantly boosts performance and speeds up pre-training compared to random initialization or simple averaging.
  • Curriculum Learning Matters: A gradual increase in task complexity during pre-training (CL) leads to better performance, especially on challenging biomedical NER tasks. This suggests that structured learning can be more effective than a uniform training approach.
  • Whole-Word Masking Isn't Always Superior: While whole-word masking (WWM) can be beneficial, it's not universally the best strategy for all biomedical NER tasks. Masking individual tokens/subtokens can sometimes yield better results.
  • Corruption Strategy Impact: The "80-10-10" corruption strategy, while standard, can negatively impact performance by increasing anisotropy in word embeddings. Removing it in later CL phases is beneficial.
  • Continued Pre-training is Efficient: Starting with a general-domain LM and continuing pre-training on biomedical data is more efficient than training from scratch, especially when combined with domain-specific vocabulary and techniques like contextualized weight distillation.

Key Results

  • The proposed model, BIOptimus 0.4, achieved state-of-the-art results on BC5-chem, NCBI-disease, and BC2GM datasets.
  • BIOptimus achieved high performance across all five NER datasets, with F1 scores reaching up to 94.1% on BC5-chem.
  • The contextualized weight distillation method, combined with CL, reduced pre-training time by at least half while improving performance.
  • An ablation study showed that removing the CL method led to a performance drop across all NER datasets, highlighting the effectiveness of the proposed curriculum.

Limitations and Future Work

  • The study focused on five NER datasets from the BLURB benchmark and did not evaluate performance on other downstream tasks. Future work could extend this evaluation.
  • The exact reason for performance drops with the "80-10-10" corruption strategy needs further investigation, potentially related to embedding anisotropy.
  • Due to computational costs, not all possible combinations of pre-training methods and techniques were explored.
  • The models were pre-trained only on English language data.

The authors suggest future research could explore other downstream tasks, delve deeper into the impact of corruption strategies, and investigate multilingual biomedical LMs.

Summary updated

BibTeX
@inproceedings{pavlova2023bioptimus,
  author = {Vera Pavlova and Mohammed Makhlouf},
  title = {{BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure