TL;DR
This paper investigates how pretrained language models (PLMs) interact with federated learning (FL) for NLP tasks. They find that PLMs might be masking the true learning happening in FL, potentially hindering genuine domain adaptation and personalization. This has implications for building equitable and robust NLP systems.
What Problem Does This Paper Address?
Federated learning (FL) is a promising approach for training machine learning models on decentralized data while preserving user privacy. When combined with powerful pretrained language models (PLMs) like BERT or BART, FL has the potential to unlock new applications in consumer-facing NLP. However, a key challenge is understanding what these models are actually learning in a federated setting, especially when data distributions across users are non-IID (not independent and identically distributed). Specifically, the authors are concerned that the strong inductive biases of PLMs might be overshadowing the learning process of FL, leading to a false sense of domain adaptation and personalization. They aim to disentangle the contributions of PLM pretraining from the actual federated learning process to better understand its limitations and potential.
Methods
The authors explore several hypotheses about the interplay between PLMs and FL. Their core methodology involves conducting experiments on various NLP tasks using different configurations of federated learning. They compare the performance of federated models with and without pretrained weights. To quantify the trade-off between generalization (performing well on unseen data) and personalization (adapting to individual client data), they introduce and utilize the $m\Delta P$ metric. This metric measures the correlation between a client's local loss and the server's generalization loss. They also investigate the impact of client size and data distribution strategies (random vs. uniform partitioning) on these trade-offs.
Datasets
The paper uses three standard NLP datasets:
- Stanford Sentiment Treebank 2 (SST-2): For text classification (sentiment analysis).
- OntoNotes v5.0: For sequence tagging (named entity recognition).
- GigaWord: For text generation (summarization).
Key Insights and Ideas
- PLMs might be masking FL's learning: The most striking finding is that pretrained weights significantly improve federated learning performance, often to the point where the performance gap between federated and centralized training shrinks considerably. However, this improvement is less pronounced when pretrained weights are not used, suggesting that PLMs might be doing the heavy lifting for domain adaptation, rather than the federated learning process itself.
- Questionable personalization: The $m\Delta P$ metric suggests that in many scenarios, the federated model with PLMs doesn't learn much from local client distributions, raising concerns about true personalization. The generalization performance might be a regularization effect from the federated setup rather than genuine adaptation.
- Uniform distribution aids generalization: Uniformly distributing data across clients tends to reduce the performance gap between federated and centralized models, indicating that balanced data distributions are beneficial for generalization in FL.
- Client size matters, but not always as expected: While client size and distribution do influence performance, the impact is less dramatic than anticipated when using pretrained models, further supporting the idea that PLMs are providing a strong baseline.
Key Results
- Reduced performance gap with PLMs: Table 1 shows that the $\Delta(\text{Perf})$ (performance gap) between federated and centralized models increases significantly when pretrained weights are not used, highlighting their crucial role. For example, on OntoNotes, the gap increases from 1.5% to 9.8% when pretrained weights are removed.
- $m\Delta P$ analysis: Figure 1 and subsequent figures show that the $m\Delta P$ values are often neutral or slightly negative, indicating that personalization doesn't necessarily lead to better generalization, and sometimes even hinders it, especially when using PLMs.
- Uniform distribution benefits: Figure 4 shows that in uniform distribution settings, the $m\Delta P$ values tend to be more stable, suggesting a better balance between personalization and generalization.
- Time-performance trade-offs: Figure 5 indicates that while discarding updates from minor clients can reduce convergence time (fewer rounds), it doesn't drastically impact performance when PLMs are used.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited scope of tasks and models: The study focuses on a few specific NLP tasks and Transformer architectures. Exploring other language models and tasks would provide a broader understanding.
- Exploration of other FL algorithms: While FedOpt is used, investigating other federated learning algorithms could reveal different dynamics.
- Understanding pretraining data: The authors suggest that a deeper understanding of the pretraining datasets and their characteristics is crucial for truly understanding domain adaptation in PLMs.
- Characterizing domains: More sophisticated methods for characterizing domains (beyond task, topic, syntax, style) are needed to fully grasp the challenges of adaptation.
Future work should focus on developing appropriate evaluation frameworks (datasets, metrics) that can better control for the influence of pretraining corpora and more accurately assess the limits of domain adaptation in PLMs within federated settings.
Summary updated