Abstract
In recent years, with the growing amount of biomedical documents, coupled with advancement in natural language processing algorithms, the research on biomedical named entity recognition (BioNER) has increased exponentially. However, BioNER research is challenging as entities in the biomedical domain are often restricted due to limited training data, can refer to multiple concepts depending on context, and rely heavily on sub-domain specific acronyms. Existing approaches often directly adopt state-of-the-art (SOTA) models trained on general corpora, which often yields unsatisfactory results. We propose BioALBERT (A Lite Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), an effective domain-specific language model trained on large-scale biomedical corpora. BioALBERT adopts a self-supervised loss focusing on inter-sentence coherence and incorporates parameter reduction techniques to lower memory consumption and increase training speed. In our experiments, BioALBERT outperformed comparative SOTA BioNER models on eight benchmark datasets across four entity types.
The Challenge of Biomedical NER
Biomedical Named Entity Recognition (BioNER)—the task of automatically identifying entities like genes, proteins, diseases, and chemicals in text—is notoriously difficult. Unlike general English, biomedical text is filled with domain-specific acronyms, and a single term (like "BRCA1") can refer to a gene, a protein, or a disease depending entirely on the context.
While models like BERT and BioBERT have pushed the boundaries of performance, they carry significant computational overhead. These models possess hundreds of millions of parameters, requiring massive memory and long training times. Furthermore, they often struggle with inter-sentence coherence, which is vital for understanding complex medical narratives.
Methodology: Adapting ALBERT for Medicine
BioALBERT adapts the "A Lite BERT" (ALBERT) architecture specifically for the medical domain. The model utilizes three key innovations to remain lean while improving performance:
- Factorized Embedding Parameterization: By decomposing the embedding matrix into two smaller matrices, the model can increase its hidden layer size (internal capacity) without a proportional explosion in parameter count.
- Cross-Layer Parameter Sharing: BioALBERT reuses parameters across transformer layers, slashing the parameter count from 110 million (in standard BERT-base) to just 31 million.
- Sentence-Order-Prediction (SOP): Unlike the original BERT's Next-Sentence-Prediction, BioALBERT uses SOP to focus on logical flow. It learns to distinguish the correct order of consecutive sentences, improving its grasp of discourse-level coherence.
Experimental Setup
The researchers produced four variants of the model, pre-training them on massive scales of biomedical literature:
- PubMed: ~4.5 billion words.
- PMC: ~13.5 billion words.
For evaluation, BioALBERT was tested against eight benchmark BioNER datasets covering four primary entity types: Disease (NCBI-Disease, BC5CDR), Drug/Chemical (BC5CDR, BC4CHEMD), Gene/Protein (BC2GM, JNLPBA), and Species (LINNAEUS, Species-800).
Key Results and Performance Gains
BioALBERT established new state-of-the-art benchmarks across all tested datasets. Notable improvements included a 10.63% increase in F1-score for BC5CDR-disease and a 23.71% increase for the Species-800 dataset compared to previous models.
Beyond accuracy, the efficiency gains were substantial. BioALBERT (large) is significantly faster to train than comparable BERT configurations, making it more accessible for researchers with limited hardware resources.
Key Insights
- Efficiency as a Feature: Reducing parameters (by up to 18x compared to BERT-large) did not compromise performance; instead, it improved it by reducing overfitting and increasing practical utility.
- Contextual Logic: The shift to Sentence-Order-Prediction (SOP) proved crucial for capturing the logical sequence paramount in biomedical literature.
- Domain-Specific Necessity: Training on the 18.8 billion words of PubMed and PMC was essential, as general-domain models fail to capture the specific linguistic distributions of medical science.
Conclusion and Future Work
BioALBERT represents a significant step forward in making high-performance BioNER more efficient. Future research will explore hybrid pre-training (combining general and biomedical text) and expanding the model's application to other tasks like relationship extraction and clinical question answering. The authors have made four variants of BioALBERT available to the research community to facilitate further advancements in biomedical text mining.
Summary updated