FedBFPT: An Efficient Federated Learning Framework for Bert Further Pre-training

Xin'ao Wang, Huan Li, Ke Chen, Lidan Shou

Abstract

Large language models like BERT require domain-specific "further pre-training" to excel in specialized fields such as medicine or law. However, privacy concerns and resource constraints on edge devices often make centralized or standard federated learning (FL) impractical. We present FedBFPT, a novel federated learning framework that employs Progressive Learning with Sampled Deeper Layers (PL-SDL). By training only one shallow layer at a time and utilizing a sampling technique to simulate full-model depth, FedBFPT reduces communication costs by over 92% and computation by nearly half. Experimental results across biology, computer science, and medicine domains demonstrate that FedBFPT matches or exceeds the performance of traditional FL and centralized baselines while significantly lowering resource requirements.

Introduction

Large language models (LLMs) like BERT are typically pre-trained on massive general-purpose datasets. To achieve peak performance in specialized domains, these models undergo "further pre-training" on domain-specific corpora. This process faces two major hurdles: data privacy and resource constraints. Specialized data is often siloed and sensitive, preventing centralized collection. While Federated Learning (FL) addresses privacy by keeping data on-device, standard FL requires clients to train and transmit the entire model, which is often beyond the capabilities of resource-constrained edge devices.

The FedBFPT Framework

FedBFPT introduces a strategy called Progressive Learning with Sampled Deeper Layers (PL-SDL) to make federated pre-training viable on low-resource hardware.

Progressive Learning and Local Models

Instead of updating all layers of a BERT model simultaneously, FedBFPT focuses on training one layer at a time, starting from the shallowest. Research suggests that shallower layers stabilize earlier and capture fundamental linguistic features. Clients run a reduced "local model" (e.g., 6 layers instead of 12), which drastically cuts RAM and GPU usage.

Figure 1: The overall architecture of the FedBFPT framework, illustrating the interaction between the global server and local clients through a six-step progressive learning process.

Sampled Deeper Layers (SDL)

To maintain the functional depth of the global model within a smaller local footprint, FedBFPT uses a mapping and sampling trick. The layer currently being trained is mapped directly from the global model. To fill the remaining slots in the local model, deeper layers from the global model are randomly sampled. This provides the necessary "context" of the deeper network without the overhead of training it.

Experimental Evaluation

The framework was evaluated across three specialized domains using the S2ORC (Semantic Scholar Open Research Corpus) for pre-training:

  • Biology: Named Entity Recognition (NER) on the JNLPBA dataset.
  • Computer Science: NER on the SciERC dataset.
  • Medicine: Document classification on the Rct-20k dataset.

Results and Discussion

FedBFPT demonstrates significant efficiency gains without sacrificing accuracy. Compared to standard Federated Learning (BERT-FL), FedBFPT reduced computational costs by 46.70% and slashed communication costs by approximately 93%.

Figure 4: Performance comparison of FedBFPT ("Ours") against various baselines across three domain-specific datasets: JNLPBA (Biology), SciERC (CS), and Rct-20k (Medicine).

As shown in the performance plots, FedBFPT consistently matches or outperforms standard BERT-FL and centralized training (Bert-Center). Notably, on the Rct-20k task, FedBFPT achieved an accuracy of 0.8312, surpassing both the centralized baseline (0.8121) and standard FL (0.8185). It also consistently outperformed "lite" models like DistilBERT and TinyBERT, proving that training a high-quality slice of a large model is more effective than training a smaller standalone model.

Conclusion

FedBFPT makes domain-specific BERT pre-training practical for resource-constrained environments. By focusing on progressive shallow-layer training and using clever sampling to maintain model depth, it achieves massive reductions in communication and computation costs. Future work will explore handling highly non-IID data distributions and accommodating heterogeneous hardware capabilities among clients.

Summary updated

Abstract

This study proposes FEDBFPT (Federated BERT Further Pre-Training), a Federated Learning (FL) framework for further pre-training the BERT language model in specialized domains while addressing privacy concerns. FEDBFPT enables multiple clients to collaboratively train the shallower layers of BERT, which are crucial in the pre-training stage, without the need to share private data. To achieve this, FEDBFPT involves building a local model for each client, progressively training the shallower layers of local models while sampling deeper layers, and aggregating trained parameters on a server to create the final global model. This approach utilizes multiple smaller local models to further pre-train a global model targeted at specific tasks via fine-tuning, resulting in a reduction in resource usage while maintaining model accuracy. Theoretical analysis is conducted to support the efficiency of FEDBFPT, and experiments are conducted on corpora across domains such as medicine, biology, and computer science. Results indicate that FEDBFPT achieves performance levels comparable to traditional FL methods while reducing computation and communication costs by 46.70% and 7.04%, respectively, even approaching the performance of centralized training models. The Source code is released at https://github.com/Hanzhouu/FedBFPT.
BibTeX
@inproceedings{wang2023fedbfpt,
  author = {Xin'ao Wang and Huan Li and Ke Chen and Lidan Shou},
  title = {{FedBFPT: An Efficient Federated Learning Framework for Bert Further Pre-training}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/483},
}
Figure