BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

Usman Naseem, Matloob Khushi, Vinay Reddy, Sakthivel Rajendran, Imran Razzak, Jinman Kim

TL;DR

This paper introduces BioALBERT, a domain-specific pre-trained language model for Biomedical Named Entity Recognition (BioNER). By adapting ALBERT's efficient architecture and training it on a massive biomedical corpus, BioALBERT significantly outperforms existing models on various BioNER tasks, addressing challenges like limited data and context-dependent entity meanings.

What Problem Does This Paper Address?

The biomedical field is generating an ever-increasing volume of text data, making it crucial to develop effective tools for extracting valuable information. Biomedical Named Entity Recognition (BioNER) is a fundamental task that aims to identify and classify entities like diseases, genes, and chemicals within this literature. However, BioNER faces several hurdles:

  • Limited Training Data: Acquiring large, annotated datasets for specific biomedical tasks is challenging and expensive.
  • Contextual Ambiguity: A single term can refer to different entities or concepts depending on its surrounding text (e.g., "BRCA1" can be a gene or a disease).
  • Acronym Proliferation: The heavy use of domain-specific acronyms (e.g., "RA") adds complexity to accurate entity identification.

Existing approaches often adapt general-purpose language models (like BERT) to the biomedical domain, but these models can struggle with the unique challenges of this field, leading to suboptimal performance.

Methods

The authors propose BioALBERT, a novel pre-trained language model specifically designed for BioNER. Their approach builds upon the ALBERT architecture, known for its parameter-efficient design, and adapts it for the biomedical domain. Key methodological contributions include:

  • Domain-Specific Pre-training: BioALBERT is pre-trained on a massive corpus of biomedical texts, including PubMed abstracts and PMC full-text articles. This allows the model to learn domain-specific language patterns and context.
  • ALBERT Architecture Adaptation: BioALBERT leverages ALBERT's core innovations:
  • Cross-layer Parameter Sharing: This technique significantly reduces the number of parameters by reusing network layers, making the model more memory-efficient and faster to train.
  • Factorized Embedding Parameterization: The large vocabulary embedding matrix is decomposed into two smaller matrices, further reducing parameter count without sacrificing performance.
  • Sentence-Order Prediction (SOP): Instead of BERT's Next Sentence Prediction (NSP), BioALBERT uses SOP, which focuses on inter-sentence coherence, helping the model learn better contextual representations.
  • Fine-tuning for BioNER: After pre-training, BioALBERT is fine-tuned on specific BioNER datasets. This process adapts the general biomedical knowledge learned during pre-training to the specific task of entity recognition.

Datasets

The paper evaluates BioALBERT on eight biomedical NER benchmark datasets, covering four main entity types:

  • Disease: NCBI Disease, BC5CDR (Disease sub-task)
  • Drug/Chemical: BC5CDR (Chemical sub-task), BC4CHEMD
  • Drug/Protein: BC2GN, JNLPBA
  • Species: LINNAEUS, Species-800

These datasets are publicly available and represent a diverse range of biomedical entity recognition tasks. The authors did not introduce new datasets but utilized existing, well-established benchmarks.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: The study strongly suggests that pre-training language models on domain-specific corpora (like biomedical texts) is essential for achieving state-of-the-art performance in that domain, rather than relying solely on general-domain pre-trained models.
  • ALBERT's Efficiency Translates to BioNER: ALBERT's architectural innovations, particularly parameter sharing and factorized embeddings, make it a more efficient and effective base for biomedical models like BioALBERT, leading to faster training and reduced memory footprint.
  • Context Matters Deeply: The success of BioALBERT highlights the importance of capturing contextual nuances in biomedical text, especially given the prevalence of ambiguous terms and acronyms. The SOP loss function plays a key role in this.
  • Duplication Data Can Be Beneficial: The analysis of different BioALBERT variants suggests that training with duplicated data (e.g., using a "dup-factor") can sometimes lead to improved performance, indicating the potential benefits of data augmentation strategies.

Key Results

BioALBERT demonstrated superior performance across all eight BioNER datasets and entity types when compared to existing state-of-the-art models, including BioBERT.

  • Performance Gains: BioALBERT achieved significant improvements, with notable gains of up to 23.71% in F1-score on certain datasets and entity types.
  • Efficiency: BioALBERT models were trained significantly faster than their BioBERT counterparts, showcasing the benefits of ALBERT's parameter-efficient design.
  • Robustness: The paper presented four variants of BioALBERT, all of which performed competitively, indicating the robustness of the approach.

Limitations and Future Work

  • Limitations:
    • While BioALBERT shows excellent performance, the paper acknowledges that the biomedical domain is vast, and further exploration with more diverse corpora and training steps could yield even better results.
  • The paper focuses on BioNER; its effectiveness on other biomedical NLP tasks is not extensively explored.
  • Future Work:
    • The authors plan to pre-train other versions of BioALBERT using hybrid general and biomedical corpora, with more extensive training.
  • They also intend to fine-tune BioALBERT on other biomedical text mining tasks to further demonstrate its versatility and effectiveness.
  • The pre-trained BioALBERT models and source code will be made publicly available to facilitate further research.

Summary updated

Abstract

In recent years, with the growing amount of biomedical documents, coupled with advancement in natural language processing algorithms, the research on biomedical named entity recognition (BioNER) has increased exponentially. However, BioNER research is challenging as NER in the biomedical domain are: (i) often restricted due to limited amount of training data, (ii) an entity can refer to multiple types and concepts depending on its context and, (iii) heavy reliance on acronyms that are sub-domain specific. Existing BioNER approaches often neglect these issues and directly adopt the state-of-the-art (SOTA) models trained in general corpora, which often yields unsatisfactory results. We propose biomedical ALBERT (A Lite Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) - bioALBERT - an effective domain-specific pre-trained language model trained on a huge biomedical corpus designed to capture biomedical context-dependent NER. We adopted a self-supervised loss function used in ALBERT that targets modelling inter-sentence coherence to better learn context-dependent representations and incorporated parameter reduction strategies to minimise memory usage and enhance the training time in BioNER. In our experiments, BioALBERT outperformed comparative SOTA BioNER models on 8 biomedical NER benchmark datasets with 4 different entity types. The performance is increased for; (i) disease type corpora by 7.47% (NCBI- disease) and 10.63% (BC5CDR-disease); (ii) drug-chem type corpora by 4.61 % (BC5CDR-Chem) and 3.89% (BC4CHEMD); (iii) gene-protein type corpora by 12.25% (BC2GM) and 6.42% (JNLPBA); and (iv) species type corpora by 6.19% (LINNAEUS) and 23.71 % (Species-800) is observed which leads to a state-of-the-art results. The performance of a proposed model on four different biomedical entity types shows that our model is robust and generalisable in recognising biomedical entities in text.
BibTeX
@inproceedings{naseem2021bioalbert,
  author = {Usman Naseem and Matloob Khushi and Vinay Reddy and Sakthivel Rajendran and Imran Razzak and Jinman Kim},
  title = {{BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition}},
  booktitle = {IJCNN},
  year = {2021},
  doi = {10.1109/ijcnn52387.2021.9533884},
}
Figure