Abstract
Large language models like BERT require domain-specific "further pre-training" to excel in specialized fields such as medicine or law. However, privacy concerns and resource constraints on edge devices often make centralized or standard federated learning (FL) impractical. We present FedBFPT, a novel federated learning framework that employs Progressive Learning with Sampled Deeper Layers (PL-SDL). By training only one shallow layer at a time and utilizing a sampling technique to simulate full-model depth, FedBFPT reduces communication costs by over 92% and computation by nearly half. Experimental results across biology, computer science, and medicine domains demonstrate that FedBFPT matches or exceeds the performance of traditional FL and centralized baselines while significantly lowering resource requirements.
Introduction
Large language models (LLMs) like BERT are typically pre-trained on massive general-purpose datasets. To achieve peak performance in specialized domains, these models undergo "further pre-training" on domain-specific corpora. This process faces two major hurdles: data privacy and resource constraints. Specialized data is often siloed and sensitive, preventing centralized collection. While Federated Learning (FL) addresses privacy by keeping data on-device, standard FL requires clients to train and transmit the entire model, which is often beyond the capabilities of resource-constrained edge devices.
The FedBFPT Framework
FedBFPT introduces a strategy called Progressive Learning with Sampled Deeper Layers (PL-SDL) to make federated pre-training viable on low-resource hardware.
Progressive Learning and Local Models
Instead of updating all layers of a BERT model simultaneously, FedBFPT focuses on training one layer at a time, starting from the shallowest. Research suggests that shallower layers stabilize earlier and capture fundamental linguistic features. Clients run a reduced "local model" (e.g., 6 layers instead of 12), which drastically cuts RAM and GPU usage.
Sampled Deeper Layers (SDL)
To maintain the functional depth of the global model within a smaller local footprint, FedBFPT uses a mapping and sampling trick. The layer currently being trained is mapped directly from the global model. To fill the remaining slots in the local model, deeper layers from the global model are randomly sampled. This provides the necessary "context" of the deeper network without the overhead of training it.
Experimental Evaluation
The framework was evaluated across three specialized domains using the S2ORC (Semantic Scholar Open Research Corpus) for pre-training:
- Biology: Named Entity Recognition (NER) on the JNLPBA dataset.
- Computer Science: NER on the SciERC dataset.
- Medicine: Document classification on the Rct-20k dataset.
Results and Discussion
FedBFPT demonstrates significant efficiency gains without sacrificing accuracy. Compared to standard Federated Learning (BERT-FL), FedBFPT reduced computational costs by 46.70% and slashed communication costs by approximately 93%.
As shown in the performance plots, FedBFPT consistently matches or outperforms standard BERT-FL and centralized training (Bert-Center). Notably, on the Rct-20k task, FedBFPT achieved an accuracy of 0.8312, surpassing both the centralized baseline (0.8121) and standard FL (0.8185). It also consistently outperformed "lite" models like DistilBERT and TinyBERT, proving that training a high-quality slice of a large model is more effective than training a smaller standalone model.
Conclusion
FedBFPT makes domain-specific BERT pre-training practical for resource-constrained environments. By focusing on progressive shallow-layer training and using clever sampling to maintain model depth, it achieves massive reductions in communication and computation costs. Future work will explore handling highly non-IID data distributions and accommodating heterogeneous hardware capabilities among clients.
Summary updated