BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba

Ling Yue, Sixue Xing, Yingzhou Lu, Tianfan FuarXiv 2024 · arXiv · OpenReview

Abstract

The advancement of natural language processing (NLP) in biology hinges on models' ability to interpret intricate biomedical literature. Traditional models often struggle with the complex and domain-specific language in this field. In this paper, we present BioMamba, a pre-trained model specifically designed for biomedical text mining. BioMamba builds upon the Mamba architecture and is pre-trained on an extensive corpus of biomedical literature. Our empirical studies demonstrate that BioMamba significantly outperforms models like BioBERT and general-domain Mamba across various biomedical tasks. For instance, BioMamba achieves a 100 times reduction in perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test set. We provide an overview of the model architecture, pre-training process, and fine-tuning techniques. Additionally, we release the code and trained model to facilitate further research.

Introduction

The biomedical field generates an exponential amount of literature, but extracting actionable insights remains a significant challenge. Traditional Natural Language Processing (NLP) models, primarily based on the Transformer architecture (such as BERT and GPT), encounter two primary hurdles in this domain:

  1. Computational Inefficiency: Transformers rely on a self-attention mechanism with quadratic complexity ($O(n^2)$). As document length increases, memory and processing requirements grow exponentially, making long-sequence modeling prohibitively expensive.
  2. Domain Specificity: General-purpose models often fail to grasp the intricate, highly technical terminology found in medical journals. This lack of domain knowledge can lead to "hallucinations" or irrelevant outputs when applied to clinical data.

To address these limitations, we introduce BioMamba, a model designed to be both computationally efficient and deeply specialized in biomedical science.

Methodology

BioMamba departs from the standard Transformer design by utilizing the Mamba architecture, which is built on Structured State Space Models (SSMs).

Architectural Advantages

Unlike Transformers, Mamba scales linearly ($O(n)$) with sequence length. This efficiency makes it significantly faster and less resource-intensive for processing long biomedical texts. Furthermore, Mamba employs a selective mechanism with dynamic parameters that allow the model to selectively propagate or forget information based on the current input. This mimics the recurrent nature of RNNs while maintaining the parallel processing capabilities of Transformers.

Comparison of traditional RNN and Transformer architectures.

Training Strategy

The development of BioMamba followed a two-stage process. The model was first initialized with weights from the general-purpose Mamba-130m model. Subsequently, we performed extensive domain-specific pre-training on a massive corpus of PubMed abstracts, totaling approximately 2.5 billion tokens. This stage was crucial for bridging the "domain gap" between general web text and specialized biomedical literature.

Datasets and Evaluation

BioMamba was evaluated using several key benchmarks:

  • Pre-training Corpus: PubMed, the gold standard database for biomedical literature.
  • Downstream Tasks: The BioASQ factoid datasets (versions 4b, 5b, and 6b). These benchmarks require models to extract precise answers from clinical text, serving as a rigorous test for biomedical question answering (QA).
Comparison of BERT's bidirectional masking vs. Mamba's autoregressive approach, alongside the PubMed and BioASQ datasets.

Key Results

BioMamba demonstrated dominant performance across all evaluated metrics:

  • Language Modeling Efficiency: On PubMed data, BioMamba achieved a 100x reduction in perplexity (dropping from 505.62 to 2.93) and a 4x reduction in cross-entropy loss compared to the base Mamba model.
  • Question Answering Accuracy: On the BioASQ 4b dataset, BioMamba achieved an Accuracy (ACC) of 0.359, more than doubling the performance of both BioBERT (0.154) and BioGPT (0.154).
  • Consistency: BioMamba consistently outperformed all baselines across the 4b, 5b, and 6b versions of the BioASQ datasets in both Accuracy and Mean Reciprocal Rank (MRR).

Discussion and Future Work

Our results underscore that architecture alone is insufficient for high-stakes fields like medicine; domain-specific data is mandatory. BioMamba proves that the SSM-based approach can capture long-range dependencies in medical text as effectively as Transformers, without the quadratic overhead.

Future research will focus on scaling the model to larger parameter versions (e.g., 2.8B or 7B) and evaluating it on a broader range of tasks, including Named Entity Recognition (NER) and relation extraction. Ultimately, we envision BioMamba powering clinical decision support systems in real-world medical environments.

Summary updated

Abstract

The advancement of natural language processing (NLP) in biology hinges on models' ability to interpret intricate biomedical literature. Traditional models often struggle with the complex and domain-specific language in this field. In this paper, we present BioMamba, a pre-trained model specifically designed for biomedical text mining. BioMamba builds upon the Mamba architecture and is pre-trained on an extensive corpus of biomedical literature. Our empirical studies demonstrate that BioMamba significantly outperforms models like BioBERT and general-domain Mamba across various biomedical tasks. For instance, BioMamba achieves a 100 times reduction in perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test set. We provide an overview of the model architecture, pre-training process, and fine-tuning techniques. Additionally, we release the code and trained model to facilitate further research.
BibTeX
@misc{yue2024biomamba,
  author = {Ling Yue and Sixue Xing and Yingzhou Lu and Tianfan Fu},
  title = {{BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba}},
  year = {2024},
  eprint = {2408.02600},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.02600},
}
Figure