TL;DR
This paper introduces BioELECTRA, a new biomedical language model that adapts the ELECTRA architecture for the domain. BioELECTRA achieves state-of-the-art results on 13 biomedical datasets and 4 clinical datasets, outperforming previous models by leveraging domain-specific pretraining.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has seen significant advancements with the development of pre-trained language models like BERT. These models, trained on general text, have been adapted for specialized domains like biomedicine through "continual pretraining" (further training on domain-specific data). However, recent research suggests that pretraining from scratch using domain-specific corpora and vocabularies can yield even better performance. The challenge lies in effectively adapting powerful general-domain architectures like ELECTRA, which uses a novel "replaced token detection" pretraining task, to the nuances of biomedical text and achieving state-of-the-art results on various biomedical NLP tasks.
Methods
The core of this work is the adaptation of the ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) architecture for the biomedical domain, resulting in BioELECTRA. ELECTRA's unique pretraining approach involves a "generator" and a "discriminator." The generator replaces some tokens in the input with plausible alternatives, and the discriminator's task is to identify which tokens were replaced. This "replaced token detection" is more sample-efficient than BERT's masked language modeling.
BioELECTRA follows a "pretraining from scratch" approach, meaning it doesn't start with general-domain BERT weights. Instead, it uses a domain-specific vocabulary and is pretrained on large biomedical text corpora: PubMed abstracts and PubMed Central (PMC) full-text articles. The authors experimented with four variants:
- BioELECTRA(P): Pretrained on PubMed abstracts.
- BioELECTRA(P+F): Pretrained on both PubMed abstracts and PMC full text.
- BioELECTRA(P+F) LT: The (P+F) model with longer training.
- BioELECTRA(W+P): Pretrained from scratch on Wikipedia and PubMed abstracts, using a general domain vocabulary.
For fine-tuning on downstream tasks, the authors followed ELECTRA's approach, using only the discriminator and applying discriminative finetuning with a learning rate decay factor.
Datasets
The paper utilizes a comprehensive set of datasets for evaluation, categorized under two main benchmarks:
- BLURB (Biomedical Language Understanding & Reasoning Benchmark): This benchmark comprises 13 biomedical datasets covering tasks like Named Entity Recognition (NER), PICO extraction, Relation Extraction (RE), Sentence Similarity, Document Classification, and Question Answering (QA).
- BLUE (Biomedical Language Understanding Evaluation) Benchmark: This benchmark includes 4 clinical datasets, also covering various NLP tasks.
The authors specifically mention using datasets like BC5-chem, BC5-disease, NCBI-disease, BC2GM, JNLPBA, EBM PICO, ChemProt, DDI, GAD, BIOSSES, HOC, PubMedQA, BioASQ, MedNLI, i2b2-2010, ShARe/CLEFE, and ClinicalSTS.
Key Insights and Ideas
- Pretraining from scratch is superior: The results strongly suggest that pretraining a language model from scratch using domain-specific text and vocabulary yields better performance than continual pretraining or using general domain vocabularies.
- ELECTRA's efficiency translates to biomedicine: The sample-efficient "replaced token detection" pretraining task of ELECTRA proves effective in the biomedical domain, allowing BioELECTRA to achieve strong results.
- Domain-specific vocabulary matters: Using a vocabulary derived from biomedical text (like that from PubMedBERT) is crucial for effectively representing specialized terms.
- Full text data is beneficial: Pretraining on full-text articles (PMC) in addition to abstracts (PubMed) shows improvements, particularly on certain tasks.
Key Results
BioELECTRA achieved state-of-the-art (SOTA) results on all 13 datasets in the BLURB benchmark and all 4 clinical datasets in the BLUE benchmark.
- BLURB Benchmark: BioELECTRA(P) achieved a BLURB score of 82.40%, a 1.3% improvement over the previous SOTA (PubMedBERT's 81.10%).
- Clinical Datasets (BLUE): BioELECTRA models achieved new SOTA on three of the four clinical datasets: i2b2-2010, ShARe/CLEFE, and ClinicalSTS.
- Specific Task Improvements:
- New SOTA of 86.34% on MedNLI (a 1.39% accuracy improvement).
- New SOTA of 64% on PubMedQA (a 2.98% accuracy improvement).
- Significantly higher scores on Question Answering tasks compared to other BERT-based models.
Limitations and Future Work
- Limitations: The paper notes that they could not compare their results with Bio-lm (Lewis et al., 2020) for all tasks due to a lack of publicly available train/test/dev split details for their datasets.
- Future Work: The authors plan to extend their domain-specific pretraining approach to larger ELECTRA models (ELECTRA-LARGE). They also intend to train ELECTRA-BASE and ELECTRA-LARGE models on MIMIC III clinical notes and evaluate their performance. Additionally, they aim to focus on Biomedical Question Answering tasks, leveraging ELECTRA's strong performance on general QA benchmarks.
Summary updated