BERT WEAVER: Using WEight AVERaging to Enable Lifelong Learning for Transformer-based Models

Lisa Langnickel, Alexander Schulz, Barbara Hammer, Juliane Fluck

Abstract

Recent developments in transfer learning have boosted the advancements in natural language processing tasks. The performance is, however, dependent on high-quality, manually annotated training data. Especially in the biomedical domain, it has been shown that one training corpus is not enough to learn generic models that are able to efficiently predict on new data. Therefore, state-of-the-art models need the ability of lifelong learning in order to improve performance as soon as new data are available - without the need of re-training the whole model from scratch. We present WEAVER, a simple, yet efficient post-processing method that infuses old knowledge into the new model, thereby reducing catastrophic forgetting. We show that applying WEAVER in a sequential manner results in similar word embedding distributions as doing a combined training on all data at once, while being computationally more efficient. Because there is no need of data sharing, the presented method is also easily applicable to federated learning settings and can for example be beneficial for the mining of electronic health records from different clinics.

Introduction

In the biomedical domain, Natural Language Processing (NLP) models—specifically for Named Entity Recognition (NER)—often struggle to generalize across different datasets. A model trained on one corpus frequently fails on another due to variations in annotation guidelines or domain-specific vocabulary.

Performance heatmaps across different training and test sets for (a) Diseases, (b) Genes/Proteins, and (c) Chemicals, illustrating the challenge of cross-dataset generalization.

To address this, models require "lifelong learning" capabilities to incorporate new data as it becomes available. However, standard fine-tuning on new data typically results in catastrophic forgetting, where the model loses previously acquired knowledge. Existing mitigation strategies have significant drawbacks: rehearsal methods violate privacy by requiring old data storage; architectural methods like Adapters increase complexity and inference time; and regularization methods like Elastic Weight Consolidation (EWC) are computationally intensive.

Methodology: The WEAVER Approach

The authors propose WEAVER (WEight AVERaging), a lightweight post-processing technique inspired by the Federated Averaging (FedAvg) algorithm. While FedAvg is designed for spatially distributed data, WEAVER adapts it for temporally distributed data arriving sequentially.

The WEAVER workflow: sequential fine-tuning followed by proportional weight averaging to combine models.

The process involves:

  1. Initialization: Fine-tune a pre-trained BioBERT model on an initial dataset.
  2. Sequential Learning: Upon receiving a new dataset, the existing model is fine-tuned further on the new data.
  3. Weight Averaging: The weights of the previous model and the newly fine-tuned model are averaged. This averaging is weighted proportionally based on the number of training examples in each respective dataset.
  4. Iteration: This cycle repeats as new data arrives, maintaining a constant architecture without storing historical data.

Experimental Setup

WEAVER was evaluated across three distinct biomedical NER tasks, using a variety of datasets to simulate a continual learning environment:

  • Diseases: 5 datasets including NCBI, BC5CDR, and miRNA-disease.
  • Proteins/Genes: 6 datasets including BioNLP11-ID and JNLPBA.
  • Chemicals: 5 datasets including BC4CHEMD and BC5CDR.

Key Results and Insights

WEAVER demonstrated superior performance compared to standard fine-tuning and established continual learning methods like EWC and AdapterFusion.

  • Performance Gains: In Disease NER, WEAVER achieved an average F1-score of 77.33%, outperforming standard FineTuning (76.59%) and EWC (76.15%). It approached the performance of Multi-Task Learning (79.50%), which serves as an upper bound requiring simultaneous access to all data.
  • Knowledge Retention: In Protein/Gene NER, WEAVER (75.26% F1) outperformed the Replay method (67.91%) and stayed within 1% of the MTL upper bound (75.98%).
  • Backward Transfer: WEAVER often exhibited positive backward transfer, where learning from new datasets improved performance on previously seen tasks.
  • Embedding Alignment: UMAP visualizations revealed that WEAVER's sequential averaging produces a word embedding distribution nearly identical to models trained on all data at once, validating its efficiency.

Conclusion and Future Work

WEAVER provides a simple yet effective defense against catastrophic forgetting. Its primary limitation is a potential bias toward larger datasets due to the proportional weighting scheme. Future research will explore dynamic weight adjustment through data drift detection and extending the method to other NLP tasks like document classification. WEAVER is currently being integrated into production systems for COVID-19 semantic search.

Summary updated

Abstract

Recent developments in transfer learning have boosted the advancements in natural language processing tasks. The performance is, however, dependent on high-quality, manually annotated training data. Especially in the biomedical domain, it has been shown that one training corpus is not enough to learn generic models that are able to efficiently predict on new data. Therefore, state-of-the-art models need the ability of lifelong learning in order to improve performance as soon as new data are available - without the need of re-training the whole model from scratch. We present WEAVER, a simple, yet efficient post-processing method that infuses old knowledge into the new model, thereby reducing catastrophic forgetting. We show that applying WEAVER in a sequential manner results in similar word embedding distributions as doing a combined training on all data at once, while being computationally more efficient. Because there is no need of data sharing, the presented method is also easily applicable to federated learning settings and can for example be beneficial for the mining of electronic health records from different clinics.
BibTeX
@misc{langnickel2022bert,
  author = {Lisa Langnickel and Alexander Schulz and Barbara Hammer and Juliane Fluck},
  title = {{BERT WEAVER: Using WEight AVERaging to Enable Lifelong Learning for Transformer-based Models}},
  year = {2022},
  eprint = {2202.10101},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.10101},
}
Figure