Scaling Federated Learning for Fine-tuning of Large Language Models

Agrin Hilmkil, Sebastian Callh, Matteo Barbieri, Leon René Sütfeld, Edvin Listo Zec, Olof Mogren

Abstract: Federated learning (FL) is a promising approach to distributed compute, as well as distributed data, and provides a level of privacy and compliance to legal frameworks. This makes FL attractive for both consumer and healthcare applications. While the area is actively being explored, few studies have examined FL in the context of larger language models and there is a lack of comprehensive reviews of robustness across tasks, architectures, numbers of clients, and other relevant factors. In this paper, we explore the fine-tuning of Transformer-based language models in a federated learning setting. We evaluate three popular BERT-variants of different sizes (BERT, ALBERT, and DistilBERT) on a number of text classification tasks such as sentiment analysis and author identification. We perform an extensive sweep over the number of clients, ranging up to 32, to evaluate the impact of distributed compute on task performance in the federated averaging setting. While our findings suggest that the large sizes of the evaluated models are not generally prohibitive to federated training, we found that the different models handle federated averaging to a varying degree. Most notably, DistilBERT converges significantly slower with larger numbers of clients, and under some circumstances, even collapses to chance level performance. Investigating this issue presents an interesting perspective for future research.

Introduction

Federated Learning (FL) allows models to be trained across multiple decentralized devices (clients) without ever sharing raw data, making it a potential solution for privacy-sensitive fields like healthcare. However, most FL research has focused on small models. As Large Language Models (LLMs) like BERT have become the standard for NLP, a critical question arises: Can we actually fine-tune these massive architectures using standard federated averaging (FedAvg) as we increase the number of participants? This study specifically examines "cross-silo" scenarios, where a few dozen organizations collaborate, rather than the "cross-device" scenario of millions of smartphones.

Methodology

The study employed the standard Federated Averaging (FedAvg) algorithm. In this setup, a central server sends a global model to several clients. Each client trains the model on its local data for a few epochs and sends the updated weights back. The server averages these weights to create a new global model, and the cycle repeats.

To evaluate the impact of architecture on performance, three variants were compared:

  • BERT-base: 110M parameters, 12 layers.
  • ALBERT-base: 11M parameters, 12 layers (uses parameter sharing).
  • DistilBERT: 65M parameters, 6 layers (created via knowledge distillation).

Experiments were conducted on three standard text classification benchmarks: IMDB (sentiment analysis), Yelp F (5-star rating classification), and AG News (news topic classification). The total amount of data and the total compute budget (number of rounds) were kept fixed, splitting the data into smaller chunks as the number of clients increased from 1 to 32.

Results and Analysis

Scaling and Robustness

Surprisingly, the massive size of BERT is not a dealbreaker for federated training. Both BERT and ALBERT showed only a gradual, manageable decline in accuracy as the number of clients increased. However, DistilBERT performed significantly worse than its peers in the FL setting. While it works well in centralized training, it showed a steep decline in accuracy when moving past 4 clients, sometimes failing to learn anything better than a random guess.

Accuracy across different numbers of clients for AG News, IMDB, and Yelp F datasets.

Convergence Dynamics

As the number of clients increased, models generally required more communication rounds to reach their performance potential. The training curves reveal that while BERT and ALBERT eventually converge to high accuracy even with 32 clients, DistilBERT's performance often flatlines at the level of a random classifier (chance level) on datasets like IMDB and Yelp F when the client count is high.

Training curves showing accuracy over rounds for BERT, ALBERT, and DistilBERT across various client counts.

Key Insights

  • Depth vs. Parameters: ALBERT (12 layers, 11M params) performed much better than DistilBERT (6 layers, 65M params). This suggests that for FL, the number of layers (depth) or the specific pre-training method might be more important for stability than the raw parameter count.
  • Task Sensitivity: Scaling was more challenging on the IMDB dataset than on AG News. The high variability in movie reviews likely creates larger differences between client models, making the weight averaging process more difficult.

Conclusion and Future Work

The study demonstrates that full-sized BERT and parameter-efficient ALBERT scale gracefully up to 32 clients in a cross-silo FL setting. The failure of DistilBERT presents a mystery; the distillation process may make its weight space less compatible with averaging. Future work should investigate whether specific hyperparameter tuning can rescue DistilBERT and how these models behave with non-I.I.D. data, which is more representative of real-world federated environments.

Summary updated

Abstract

Federated learning (FL) is a promising approach to distributed compute, as well as distributed data, and provides a level of privacy and compliance to legal frameworks. This makes FL attractive for both consumer and healthcare applications. While the area is actively being explored, few studies have examined FL in the context of larger language models and there is a lack of comprehensive reviews of robustness across tasks, architectures, numbers of clients, and other relevant factors. In this paper, we explore the fine-tuning of Transformer-based language models in a federated learning setting. We evaluate three popular BERT-variants of different sizes (BERT, ALBERT, and DistilBERT) on a number of text classification tasks such as sentiment analysis and author identification. We perform an extensive sweep over the number of clients, ranging up to 32, to evaluate the impact of distributed compute on task performance in the federated averaging setting. While our findings suggest that the large sizes of the evaluated models are not generally prohibitive to federated training, we found that the different models handle federated averaging to a varying degree. Most notably, DistilBERT converges significantly slower with larger numbers of clients, and under some circumstances, even collapses to chance level performance. Investigating this issue presents an interesting perspective for future research.
BibTeX
@inproceedings{hilmkil2021scaling,
  author = {Agrin Hilmkil and Sebastian Callh and Matteo Barbieri and Leon René Sütfeld and Edvin Listo Zec and Olof Mogren},
  title = {{Scaling Federated Learning for Fine-tuning of Large Language Models}},
  booktitle = {NLDB},
  year = {2021},
  url = {https://openreview.net/forum?id=pNOcuPWER9},
}
Figure