AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model

Yongyu Yan, Kui Xue, Xiaoming Shi, Qi Ye, Jingping Liu, Tong Ruan

TL;DR

The AF Adapter (Attention-FFN Adapter) is a novel continual pretraining method that builds high-performing domain-specific language models while mitigating "catastrophic forgetting." By adding a small number of parallel attention heads and hidden units to a frozen backbone, the authors created a Chinese biomedical model that outperforms strong baselines on the CBLUE benchmark using only 17% of the original parameters.

What Problem Does This Paper Address?

Pretrained Language Models (PLMs) like BERT and RoBERTa are powerful, but they often struggle with specialized domains like biomedicine due to differences in word distribution and complex terminology.

Researchers typically use continual pretraining—taking a general-domain model and training it further on domain-specific data—to bridge this gap. However, this process often leads to catastrophic forgetting, where the model "overwrites" its general-purpose knowledge (like common sense and basic grammar) to learn the new domain. As shown in Figure 1, a standard fine-tuned model might correctly identify medical terms but start making errors on basic vocabulary it once knew.

Fig. 1. Outputs of RoBERTa, fine-tuned RoBERTa, and AF Adapter based RoBERTa for two input items.

Methods

The authors propose a "layer-extending" approach called the AF Adapter. Unlike traditional adapters that add new layers sequentially (increasing the depth of the network), the AF Adapter extends existing layers in parallel.

Key components include:

  • Extended Attention Layer: The authors add a small number of "domain-specific" attention heads alongside the original ones. The original heads are frozen, while the new ones are trained on biomedical data.
  • Extended Feed-Forward Network (FFN): Similarly, they add new hidden units to the FFN in parallel to the original structure.
  • Parameter Freezing: During continual pretraining, all original parameters from the backbone (RoBERTa-wwm-ext-base) are frozen. Only the newly added "adapter" units are trainable.

This architecture, illustrated in Figure 2, ensures that the model retains its original knowledge base while gaining specialized capacity without increasing the computational depth of the network.

The AF Adapter architecture, showing how new heads and units are added in parallel to the frozen original model.

Datasets

The model was pretrained on a massive collection of Chinese biomedical text totaling approximately 171 million tokens:

  • Biomedical Question Answering: 93M tokens.
  • Medical Encyclopedia: 33M tokens (from Baidu Encyclopedia and Wikipedia).
  • Electronic Medical Records: 45M tokens (from Fudan University Shanghai Cancer Center).

For evaluation, the authors used the CBLUE (Chinese Biomedical Language Understanding Evaluation) benchmark, which covers eight tasks across six categories: Named Entity Recognition (NER), Relationship Extraction (RE), Lexical Normalization (NORM), Textual Similarity (TS), Text Classification (TC), and Natural Language Inference (NLI).

Key Insights and Ideas

  • Parallel is Better than Sequential: By extending layers horizontally rather than adding them vertically, the AF Adapter avoids the "information forgetting" that can occur when inputs have to pass through more and more layers.
  • Efficiency vs. Performance: You don't need to retrain the whole model. Training just ~17% of the parameters (the new adapter units) actually yielded better results than full fine-tuning.
  • Stability: The AF Adapter showed a smoother, more gradual decrease in loss during training compared to other techniques like FL-Tuning, suggesting a more robust learning process for domain-specific information (see Figure 5).
Fig. 5. Convergence comparison of Fine-Tuning, FL-Tuning, LoRA and AF Adapter. The curves in dark color are obtained by smoothing the loss curves (light color). The smoothing function is α ∗previous smoothed value + (1 −α) ∗current value, where α = 0.6 is the smooth weight.

Key Results

  • Superior Performance: AF Adapter based RoBERTa achieved the highest average score (69.904%) on the CBLUE benchmark, outperforming strong baselines like PCL-MedBERT (67.9%) and MacBERT-base (69.0%).
  • Mitigating Forgetting: In a specific test for catastrophic forgetting using general-domain samples, the AF Adapter reduced the accuracy drop by 11% compared to standard fine-tuning.
  • Task-Specific Wins: The model showed particularly strong gains in Relationship Extraction (RE) and Natural Language Inference (NLI), surpassing the backbone model by 1.2% and 1.9% respectively.

Limitations and Future Work

The authors note that the AF Adapter exhibits slower convergence compared to methods like LoRA. Because it incorporates a parallel structure, the model requires more iterations to reach an optimal solution.

While the paper focuses on the Chinese biomedical domain, the authors suggest that this layer-extending approach is a generalizable strategy for building domain-specific models in other fields or languages, though specific future directions were not detailed.

Summary updated

Abstract

Continual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model. In spite of its high efficiency, continual pretraining suffers from catastrophic forgetting, which may harm the model's performance in downstream tasks. To alleviate the issue, in this paper, we propose a continual pretraining method for the BERT-based model, named Attention-FFN Adapter. Its main idea is to introduce a small number of attention heads and hidden units inside each self-attention layer and feed-forward network. Furthermore, we train a domain-specific language model named AF Adapter based RoBERTa for the Chinese biomedical domain. In experiments, models are applied to downstream tasks for evaluation. The results demonstrate that with only about 17% of model parameters trained, AF Adapter achieves 0.6%, 2% gain in performance on average, compared to strong baselines. Further experimental results show that our method alleviates the catastrophic forgetting problem by 11% compared to the fine-tuning method.
BibTeX
@misc{yan2022af,
  author = {Yongyu Yan and Kui Xue and Xiaoming Shi and Qi Ye and Jingping Liu and Tong Ruan},
  title = {{AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model}},
  year = {2022},
  eprint = {2211.11363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.11363},
}
Figure