TL;DR
This paper introduces four new large biomedical language models (BioM-BERT, BioM-ALBERT, BioM-ELECTRA) by adapting existing powerful Transformer architectures to the biomedical domain. They demonstrate that careful consideration of design choices, such as vocabulary and corpus selection, significantly impacts performance on downstream biomedical tasks, achieving state-of-the-art results with comparable or lower computational costs.
What Problem Does This Paper Address?
The explosion of biomedical literature has created a need for sophisticated language models that can understand and process this specialized text. While general-purpose models like BERT have shown promise, adapting them to the nuances of the biomedical domain is crucial. Existing biomedical models often differ in their pre-training strategies (e.g., vocabulary, corpus, training objectives), making it difficult to isolate the impact of specific design choices on performance. This paper aims to systematically investigate how these design choices influence the effectiveness of large biomedical language models across various downstream tasks.
Methods
The authors leverage three prominent Transformer architectures: BERT, ALBERT, and ELECTRA. They pre-train new biomedical versions of these models, referred to as BioM-BERT, BioM-ALBERT, and BioM-ELECTRA. Key methodological aspects include:
- Architecture Adaptation: They use the original implementations of BERT, ALBERT, and ELECTRA, adapting them to the biomedical domain.
- Domain-Specific Pre-training: They pre-train these models on biomedical corpora, primarily PubMed abstracts and PMC articles.
- Vocabulary Engineering: They experiment with both general-domain vocabularies and domain-specific vocabularies derived from biomedical text.
- Design Choice Variations: They specifically explore the impact of:
- Corpora: Using PubMed abstracts vs. PubMed + PMC articles.
- Vocabulary: General domain vs. specific biomedical vocabulary.
- Model Scale: Comparing base and large versions.
- Pre-training Objective: Leveraging ELECTRA's discriminative pre-training approach.
- Fine-tuning: The pre-trained models are then fine-tuned on a suite of downstream biomedical tasks, including Named Entity Recognition (NER), Relation Extraction (RE), and Question Answering (QA).
Datasets
- Pre-training Corpora:
- PubMed abstracts
- PMC articles
- Downstream Task Datasets:
- Named Entity Recognition (NER): BC5CDR-Chemical, BC5CDR-Disease, NCBI-Disease.
- Relation Extraction (RE): ChemProt.
- Question Answering (QA): BioASQ7B-factoid (SQuADv1.1 format), SQuAD2.0.
The paper does not introduce a new benchmark dataset but rather uses existing ones to evaluate their models.
Key Insights and Ideas
- Vocabulary Matters for NER: Using a specific domain vocabulary significantly improves performance on Named Entity Recognition tasks.
- ELECTRA's Effectiveness: The ELECTRA objective shows promise in improving performance on tasks like ChemProt, SQuAD, and BioASQ, even outperforming other models with similar design choices and less computational effort.
- Domain Sensitivity in QA: Question answering tasks are particularly sensitive to out-of-domain corpora. Introducing general domain pre-training (as in BioM-ALBERTxxlarge with a general domain vocabulary) can boost performance on general QA datasets like SQuAD but may negatively impact performance on biomedical QA tasks like BioASQ.
- Model Scale vs. Task: The performance gap between base-scale and large-scale biomedical models is smaller for NER tasks compared to Relation Extraction and Question Answering.
- Fine-tuning Efficiency: Hidden layer size (H) significantly influences fine-tuning time, with larger models taking considerably longer.
Key Results
- The proposed BioM-Transformers models achieve state-of-the-art results on several biomedical tasks, including NER and RE, despite using similar or less computational cost compared to existing models.
- For instance, BioM-BERTLarge achieved a 78.8 F1 score on ChemProt at 100K steps with a 0.4x C ratio, matching BioRoBERTa's performance with a 4.0x C ratio. At 400K steps, it significantly surpassed other large biomedical models.
- BioM-ELECTRABase outperformed RoBERTaBase on ChemProt and SQuAD tasks, despite using less computational resources.
- BioM-ELECTRALarge showed superior performance on the BC5-CDR dataset for NER tasks.
- On the SQuAD dataset, BioM-ALBERTxxlarge achieved an impressive 88.0% accuracy.
Limitations and Future Work
- Out-of-Domain QA: The paper highlights the challenge of out-of-domain corpora for QA tasks, noting that general domain pre-training can improve general QA performance (SQuAD) but decrease biomedical QA performance (BioASQ). They suggest the need for a biomedical QA task in the SQuADv2.0 format.
- Fine-tuning Checkpoints: For BioASQ, using a SQuAD fine-tuned checkpoint improved performance, suggesting that general domain fine-tuning can be beneficial. However, this approach was not used for NER and RE tasks, where the gap between general and biomedical domains was larger.
- Future Work:
- Extend evaluation to additional biomedical tasks.
- Investigate implementing early exiting techniques to reduce fine-tuning time.
- Build an end-to-end ensemble QA system using their large models and Sentence-BERT to address pandemic-related issues.
Summary updated