Scaling Federated Learning for Fine-Tuning of Large Language Models

Agrin Hilmkil, Sebastian Callh, Matteo Barbieri, Leon René Sütfeld, Edvin Listo Zec, Olof Mogren

TL;DR

This paper investigates how well large Transformer-based models (like BERT) handle being fine-tuned in a Federated Learning (FL) environment. The researchers found that while full-sized BERT and the parameter-efficient ALBERT scale gracefully up to 32 clients, the distilled version, DistilBERT, surprisingly struggles and often collapses to "chance-level" performance as the number of clients increases.

What Problem Does This Paper Address?

Federated Learning (FL) allows models to be trained across multiple decentralized devices (clients) without ever sharing raw data, making it a "holy grail" for privacy-sensitive fields like healthcare. However, most FL research has focused on small models (e.g., simple CNNs or LSTMs).

As Large Language Models (LLMs) like BERT have become the standard for NLP, a critical question arises: Can we actually fine-tune these massive architectures using standard federated averaging (FedAvg) as we increase the number of participants? The authors specifically look at "cross-silo" scenarios—where you might have a few dozen hospitals or organizations collaborating—rather than the "cross-device" scenario of millions of smartphones.

Methods

The authors employed the standard Federated Averaging (FedAvg) algorithm. In this setup, a central server sends a global model to several clients. Each client trains the model on its local data for a few epochs and sends the updated weights back. The server averages these weights to create a new global model, and the cycle repeats.

To see how architecture affects performance, they compared three variants:

  • BERT-base: The heavy hitter (110M parameters, 12 layers).
  • ALBERT-base: A "lite" version that uses parameter sharing to stay small (11M parameters, 12 layers).
  • DistilBERT: A compressed version created via knowledge distillation (65M parameters, 6 layers).

They kept the total amount of data and the total "compute budget" (number of rounds) fixed, simply splitting the data into smaller and smaller chunks as they increased the number of clients from 1 to 32.

Datasets

The study used three standard text classification benchmarks:

  1. IMDB: Binary sentiment analysis (50,000 movie reviews).
  2. Yelp F: 5-star rating classification (a much larger dataset of business reviews).
  3. AG News: 4-category news topic classification (120,000 samples).

Key Insights and Ideas

  • Size isn't the enemy: Surprisingly, the massive size of BERT isn't a dealbreaker for federated training. Both BERT and ALBERT showed only a gradual, manageable decline in accuracy as the number of clients increased.
  • The DistilBERT Mystery: DistilBERT performed significantly worse than its peers in the FL setting. While it works great in centralized training, it showed a "steep decline" in accuracy when moving past 4 clients, sometimes failing to learn anything better than a random guess.
  • Depth vs. Parameters: ALBERT (12 layers, 11M params) performed much better than DistilBERT (6 layers, 65M params). This suggests that for Federated Learning, the number of layers (depth) or the specific pre-training method might be more important for stability than the raw parameter count.
  • Task Sensitivity: Scaling was harder on the IMDB dataset than on AG News. The authors hypothesize that the high variability in movie reviews creates larger differences between client models, making the averaging process more difficult.

Key Results

  • Robustness: BERT and ALBERT maintained high accuracy even when the data was split across 32 clients.
  • Convergence Speed: As the number of clients increased, models required more communication rounds to reach 90% of their baseline (non-federated) performance.
  • Failure Modes: DistilBERT failed to reach the 90% performance threshold on the Yelp F dataset with 16 clients and on the IMDB dataset with as few as 4 clients. In several runs, DistilBERT's accuracy flatlined at the level of a random classifier.

Limitations and Future Work

  • I.I.D. Data: This study used "Independent and Identically Distributed" data, meaning every client had a similar mix of data. In the real world, data is "non-i.i.d." (e.g., one hospital might only have elderly patients), which is much harder for FL to handle.
  • Client Scale: The study stopped at 32 clients. While appropriate for "cross-silo" FL, it doesn't tell us how these models would behave on thousands of mobile devices.
  • Hyperparameter Tuning: To ensure a fair comparison, the authors used fixed hyperparameters. It’s possible that DistilBERT could be "rescued" with specific tuning for the federated setting.
  • The "Why" of DistilBERT: The authors suggest that the distillation process used to create DistilBERT might make its weight space less "friendly" to averaging, but this remains a hypothesis for future investigation.

Summary updated

Abstract

Federated learning (FL) is a promising approach to distributed compute, as well as distributed data, and provides a level of privacy and compliance to legal frameworks. This makes FL attractive for both consumer and healthcare applications. However, few studies have examined FL in the context of larger language models and there is a lack of comprehensive reviews of robustness across tasks, architectures, numbers of clients, and other relevant factors. In this paper, we explore the fine-tuning of large language models in a federated learning setting. We evaluate three popular models of different sizes (BERT, ALBERT, and DistilBERT) on a number of text classification tasks such as sentiment analysis and author identification. We perform an extensive sweep over the number of clients, ranging up to 32, to evaluate the impact of distributed compute on task performance in the federated averaging setting. While our findings suggest that the large sizes of the evaluated models are not generally prohibitive to federated training, we found that not all models handle federated averaging well. Most notably, DistilBERT converges significantly slower with larger numbers of clients, and under some circumstances, even collapses to chance level performance. Investigating this issue presents an interesting direction for future research.
BibTeX
@inproceedings{hilmkil2021scaling,
  author = {Agrin Hilmkil and Sebastian Callh and Matteo Barbieri and Leon René Sütfeld and Edvin Listo Zec and Olof Mogren},
  title = {{Scaling Federated Learning for Fine-Tuning of Large Language Models}},
  booktitle = {NLDB},
  year = {2021},
  url = {https://openreview.net/forum?id=pNOcuPWER9},
}

Loading related content...

Figure