BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

Usman Naseem, Matloob Khushi, Vinay Reddy, Sakthivel Rajendran, Imran Razzak, Jinman Kim

Abstract

In recent years, with the growing amount of biomedical documents, coupled with advancement in natural language processing algorithms, the research on biomedical named entity recognition (BioNER) has increased exponentially. However, BioNER research is challenging as entities in the biomedical domain are often restricted due to limited training data, can refer to multiple concepts depending on context, and rely heavily on sub-domain specific acronyms. Existing approaches often directly adopt state-of-the-art (SOTA) models trained on general corpora, which often yields unsatisfactory results. We propose BioALBERT (A Lite Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), an effective domain-specific language model trained on large-scale biomedical corpora. BioALBERT adopts a self-supervised loss focusing on inter-sentence coherence and incorporates parameter reduction techniques to lower memory consumption and increase training speed. In our experiments, BioALBERT outperformed comparative SOTA BioNER models on eight benchmark datasets across four entity types.

The Challenge of Biomedical NER

Biomedical Named Entity Recognition (BioNER)—the task of automatically identifying entities like genes, proteins, diseases, and chemicals in text—is notoriously difficult. Unlike general English, biomedical text is filled with domain-specific acronyms, and a single term (like "BRCA1") can refer to a gene, a protein, or a disease depending entirely on the context.

While models like BERT and BioBERT have pushed the boundaries of performance, they carry significant computational overhead. These models possess hundreds of millions of parameters, requiring massive memory and long training times. Furthermore, they often struggle with inter-sentence coherence, which is vital for understanding complex medical narratives.

Methodology: Adapting ALBERT for Medicine

BioALBERT adapts the "A Lite BERT" (ALBERT) architecture specifically for the medical domain. The model utilizes three key innovations to remain lean while improving performance:

  1. Factorized Embedding Parameterization: By decomposing the embedding matrix into two smaller matrices, the model can increase its hidden layer size (internal capacity) without a proportional explosion in parameter count.
  2. Cross-Layer Parameter Sharing: BioALBERT reuses parameters across transformer layers, slashing the parameter count from 110 million (in standard BERT-base) to just 31 million.
  3. Sentence-Order-Prediction (SOP): Unlike the original BERT's Next-Sentence-Prediction, BioALBERT uses SOP to focus on logical flow. It learns to distinguish the correct order of consecutive sentences, improving its grasp of discourse-level coherence.
The BioALBERT pre-training and fine-tuning pipeline, from large-scale corpora to specific BioNER tasks.

Experimental Setup

The researchers produced four variants of the model, pre-training them on massive scales of biomedical literature:

  • PubMed: ~4.5 billion words.
  • PMC: ~13.5 billion words.

For evaluation, BioALBERT was tested against eight benchmark BioNER datasets covering four primary entity types: Disease (NCBI-Disease, BC5CDR), Drug/Chemical (BC5CDR, BC4CHEMD), Gene/Protein (BC2GM, JNLPBA), and Species (LINNAEUS, Species-800).

Key Results and Performance Gains

BioALBERT established new state-of-the-art benchmarks across all tested datasets. Notable improvements included a 10.63% increase in F1-score for BC5CDR-disease and a 23.71% increase for the Species-800 dataset compared to previous models.

Beyond accuracy, the efficiency gains were substantial. BioALBERT (large) is significantly faster to train than comparable BERT configurations, making it more accessible for researchers with limited hardware resources.

Comparison of training time (in days) between various BioBERT and BioALBERT models, highlighting the efficiency of the ALBERT architecture.

Key Insights

  • Efficiency as a Feature: Reducing parameters (by up to 18x compared to BERT-large) did not compromise performance; instead, it improved it by reducing overfitting and increasing practical utility.
  • Contextual Logic: The shift to Sentence-Order-Prediction (SOP) proved crucial for capturing the logical sequence paramount in biomedical literature.
  • Domain-Specific Necessity: Training on the 18.8 billion words of PubMed and PMC was essential, as general-domain models fail to capture the specific linguistic distributions of medical science.

Conclusion and Future Work

BioALBERT represents a significant step forward in making high-performance BioNER more efficient. Future research will explore hybrid pre-training (combining general and biomedical text) and expanding the model's application to other tasks like relationship extraction and clinical question answering. The authors have made four variants of BioALBERT available to the research community to facilitate further advancements in biomedical text mining.

Summary updated

Abstract

In recent years, with the growing amount of biomedical documents, coupled with advancement in natural language processing algorithms, the research on biomedical named entity recognition (BioNER) has increased exponentially. However, BioNER research is challenging as NER in the biomedical domain are: (i) often restricted due to limited amount of training data, (ii) an entity can refer to multiple types and concepts depending on its context and, (iii) heavy reliance on acronyms that are sub-domain specific. Existing BioNER approaches often neglect these issues and directly adopt the state-of-the-art (SOTA) models trained in general corpora which often yields unsatisfactory results. We propose biomedical ALBERT (A Lite Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) bioALBERT, an effective domain-specific language model trained on large-scale biomedical corpora designed to capture biomedical context-dependent NER. We adopted a self-supervised loss used in ALBERT that focuses on modelling inter-sentence coherence to better learn context-dependent representations and incorporated parameter reduction techniques to lower memory consumption and increase the training speed in BioNER. In our experiments, BioALBERT outperformed comparative SOTA BioNER models on eight biomedical NER benchmark datasets with four different entity types. We trained four different variants of BioALBERT models which are available for the research community to be used in future research.
BibTeX
@inproceedings{naseem2021bioalbert,
  author = {Usman Naseem and Matloob Khushi and Vinay Reddy and Sakthivel Rajendran and Imran Razzak and Jinman Kim},
  title = {{BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition}},
  booktitle = {IJCNN},
  year = {2021},
  doi = {10.1109/ijcnn52387.2021.9533884},
}
Figure