TL;DR
The AF Adapter (Attention-FFN Adapter) is a novel continual pretraining method that builds high-performing domain-specific language models while mitigating "catastrophic forgetting." By adding a small number of parallel attention heads and hidden units to a frozen backbone, the authors created a Chinese biomedical model that outperforms strong baselines on the CBLUE benchmark using only 17% of the original parameters.
What Problem Does This Paper Address?
Pretrained Language Models (PLMs) like BERT and RoBERTa are powerful, but they often struggle with specialized domains like biomedicine due to differences in word distribution and complex terminology.
Researchers typically use continual pretraining—taking a general-domain model and training it further on domain-specific data—to bridge this gap. However, this process often leads to catastrophic forgetting, where the model "overwrites" its general-purpose knowledge (like common sense and basic grammar) to learn the new domain. As shown in Figure 1, a standard fine-tuned model might correctly identify medical terms but start making errors on basic vocabulary it once knew.
Methods
The authors propose a "layer-extending" approach called the AF Adapter. Unlike traditional adapters that add new layers sequentially (increasing the depth of the network), the AF Adapter extends existing layers in parallel.
Key components include:
- Extended Attention Layer: The authors add a small number of "domain-specific" attention heads alongside the original ones. The original heads are frozen, while the new ones are trained on biomedical data.
- Extended Feed-Forward Network (FFN): Similarly, they add new hidden units to the FFN in parallel to the original structure.
- Parameter Freezing: During continual pretraining, all original parameters from the backbone (RoBERTa-wwm-ext-base) are frozen. Only the newly added "adapter" units are trainable.
This architecture, illustrated in Figure 2, ensures that the model retains its original knowledge base while gaining specialized capacity without increasing the computational depth of the network.
Datasets
The model was pretrained on a massive collection of Chinese biomedical text totaling approximately 171 million tokens:
- Biomedical Question Answering: 93M tokens.
- Medical Encyclopedia: 33M tokens (from Baidu Encyclopedia and Wikipedia).
- Electronic Medical Records: 45M tokens (from Fudan University Shanghai Cancer Center).
For evaluation, the authors used the CBLUE (Chinese Biomedical Language Understanding Evaluation) benchmark, which covers eight tasks across six categories: Named Entity Recognition (NER), Relationship Extraction (RE), Lexical Normalization (NORM), Textual Similarity (TS), Text Classification (TC), and Natural Language Inference (NLI).
Key Insights and Ideas
- Parallel is Better than Sequential: By extending layers horizontally rather than adding them vertically, the AF Adapter avoids the "information forgetting" that can occur when inputs have to pass through more and more layers.
- Efficiency vs. Performance: You don't need to retrain the whole model. Training just ~17% of the parameters (the new adapter units) actually yielded better results than full fine-tuning.
- Stability: The AF Adapter showed a smoother, more gradual decrease in loss during training compared to other techniques like FL-Tuning, suggesting a more robust learning process for domain-specific information (see Figure 5).
Key Results
- Superior Performance: AF Adapter based RoBERTa achieved the highest average score (69.904%) on the CBLUE benchmark, outperforming strong baselines like PCL-MedBERT (67.9%) and MacBERT-base (69.0%).
- Mitigating Forgetting: In a specific test for catastrophic forgetting using general-domain samples, the AF Adapter reduced the accuracy drop by 11% compared to standard fine-tuning.
- Task-Specific Wins: The model showed particularly strong gains in Relationship Extraction (RE) and Natural Language Inference (NLI), surpassing the backbone model by 1.2% and 1.9% respectively.
Limitations and Future Work
The authors note that the AF Adapter exhibits slower convergence compared to methods like LoRA. Because it incorporates a parallel structure, the model requires more iterations to reach an optimal solution.
While the paper focuses on the Chinese biomedical domain, the authors suggest that this layer-extending approach is a generalizable strategy for building domain-specific models in other fields or languages, though specific future directions were not detailed.
Summary updated