AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model

Yongyu Yan, Kui Xue, Xiaoming Shi, Qi Ye, Jingping Liu, Tong Ruan

Abstract

Continual pretraining is a standard technique for adapting general-domain language models to specific domains. However, it often suffers from catastrophic forgetting, where the model loses its general-domain capabilities. We propose the Attention-FFN (AF) Adapter, a layer-extending method that introduces a small number of trainable attention heads and hidden units within existing layers. By freezing the backbone model and only training these adapters (approximately 17% of total parameters), we developed a Chinese biomedical RoBERTa model that outperforms strong baselines on the CBLUE benchmark and reduces catastrophic forgetting by 11% compared to standard fine-tuning.

Introduction: The Challenge of Domain Adaptation

Pretrained Language Models (PLMs) like RoBERTa are typically trained on general corpora. When applied to specialized domains such as medicine, they encounter "distribution bias" due to unique terminology and linguistic patterns. While continual pretraining on domain-specific data can bridge this gap, it frequently leads to catastrophic forgetting. In this scenario, the model's weights are updated to fit the new domain, overwriting the general-domain knowledge acquired during initial pretraining.

Illustration of catastrophic forgetting in standard fine-tuning vs. the proposed method.

Existing adapter-based methods often add new layers (layer-adding), which increases model depth and can hinder the flow of original information. The AF Adapter addresses this by extending the width of existing layers.

The AF Adapter Architecture

The core innovation of the AF Adapter is its layer-extending approach. Instead of stacking new layers, it modifies the internal structure of the Transformer block:

  1. Extended Attention (ATT): New, trainable attention heads are added in parallel to the original, frozen attention heads.
  2. Extended Feed-Forward Network (FFN): Additional hidden units are injected into the FFN, also in parallel with the frozen original units.
Architecture of the AF Adapter showing layer-extending components.

By keeping the model depth constant and freezing the backbone, the AF Adapter preserves the original feature space while allowing the new units to capture domain-specific nuances.

Experimental Setup

The model was pretrained on a large-scale Chinese biomedical corpus comprising:

  • Biomedical Q&A: 4.8 million sentences (93M tokens).
  • Medical Encyclopedias: 33M tokens from Baidu Encyclopedia and Wikipedia.
  • Electronic Medical Records (EMR): 45M tokens from Fudan University Shanghai Cancer Center.

Evaluation was conducted using the CBLUE (Chinese Biomedical Language Understanding Evaluation) benchmark, covering tasks such as Named Entity Recognition (NER) and Relationship Extraction (RE).

Key Results and Insights

  • Performance Gains: The AF Adapter-based RoBERTa achieved an average score of 69.9% on CBLUE, surpassing both general-domain models and specialized models like PCL-MedBERT.
  • Efficiency: The model achieved state-of-the-art results by training only 21 million parameters (17% of the total), making it significantly more efficient than full fine-tuning.
  • Stability and Retention: The method reduced catastrophic forgetting by 11% compared to full fine-tuning. Training curves also indicated that the AF Adapter provides a more stable convergence path than other parameter-efficient methods like LoRA.

Conclusion and Future Work

The AF Adapter demonstrates that extending layer width is a powerful alternative to increasing depth for domain adaptation. It effectively balances the need for specialized knowledge with the preservation of general-domain features. Future research will explore extending the model's vocabulary with medical-specific tokens and scaling this architecture to larger language models.

Summary updated

Abstract

Continual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model. In spite of its high efficiency, continual pretraining suffers from catastrophic forgetting, which may harm the model's performance in downstream tasks. To alleviate the issue, in this paper, we propose a continual pretraining method for the BERT-based model, named Attention-FFN Adapter. Its main idea is to introduce a small number of attention heads and hidden units inside each self-attention layer and feed-forward network. Furthermore, we train a domain-specific language model named AF Adapter based RoBERTa for the Chinese biomedical domain. In experiments, models are applied to downstream tasks for evaluation. The results demonstrate that with only about 17% of model parameters trained, AF Adapter achieves 0.6%, 2% gain in performance on average, compared to strong baselines. Further experimental results show that our method alleviates the catastrophic forgetting problem by 11% compared to the fine-tuning method.
BibTeX
@misc{yan2022af,
  author = {Yongyu Yan and Kui Xue and Xiaoming Shi and Qi Ye and Jingping Liu and Tong Ruan},
  title = {{AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model}},
  year = {2022},
  eprint = {2211.11363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.11363},
}
Figure