Salvaging Federated Learning by Local Adaptation

Tao Yu, Eugene Bagdasaryan, Vitaly ShmatikovarXiv 2020 · OpenReview

Abstract

Federated learning (FL) is designed for training on unbalanced and non-IID data across participants. However, from a local viewpoint, many participants lack the incentive to join because the federated model is often less accurate on their data than a locally trained model. This problem is exacerbated by differential privacy and robust aggregation. This paper evaluates local adaptation techniques—fine-tuning, multi-task learning, and knowledge distillation—demonstrating that they can restore the incentive to participate by ensuring the adapted federated model outperforms local-only models for nearly all users.

The Participation Incentive Problem

Federated Learning (FL) allows collaborative model training without sharing raw data, making it ideal for sensitive applications like next-word prediction on smartphones. However, a fundamental conflict exists: while FL aims to include diverse participants to improve robustness, the mechanisms used to protect privacy (Differential Privacy) and integrity (Robust Aggregation) often "average out" the unique contributions of individual users.

For many participants, especially those with high-quality or unique local data, the resulting global model is a "net loss"—it performs worse than a model they could have trained independently. If participation leads to a worse product, users have no rational incentive to join the federation.

The Impact of Privacy and Robustness

The authors demonstrate that standard privacy and robustness protections significantly degrade the model for many users. On the Reddit next-word prediction task, Differential Privacy (DP-FED) resulted in lower accuracy than local models for 21% of participants. Robust aggregation, designed to ignore outliers, was even more damaging, making the global model worse than local ones for 52% of users.

Accuracy gap ($Acc_{global} - Acc_{local}$) across different FL frameworks. Points below the zero line represent users who lose accuracy by participating in FL.

As illustrated in the scatter plots above, there is a clear negative correlation between local model quality and the benefit gained from FL. Users with the most unique data (the "head" of the distribution) are the ones most likely to be penalized by the global aggregation process, particularly under robust or private frameworks.

Salvaging FL through Local Adaptation

Rather than redesigning global aggregation, the authors propose that participants perform local adaptation on the received global model. They evaluated three primary methods:

  • Fine-tuning (FT): Re-training the global model on local data.
  • Multi-task Learning (MTL): Using techniques like Elastic Weight Consolidation (EWC) to learn local nuances without forgetting global patterns.
  • Knowledge Distillation (KD): Using the global model as a "teacher" to guide a local "student" model, preventing overfitting on small local datasets.

Key Results

Local adaptation proved to be a universal win. In the Reddit task, it improved mean accuracy by over 2%, effectively recovering the performance lost to privacy and robustness mechanisms. In image classification (CIFAR-10), gains ranged from 3% to 7%.

Crucially, adaptation restored the incentive to participate. For the ROBUST-FED setup on Reddit, where tens of thousands of users originally had no reason to join, local adaptation made the federated model better than the local-only model for 99.96% of participants.

Conclusion

Local adaptation transforms FL from a system that potentially penalizes unique users into one that provides value to everyone. By allowing users to "tweak" the global model to their specific needs, we can maintain the privacy and robustness of the federation while ensuring every participant receives a superior model.

Summary updated

Abstract

Federated learning (FL) is a heavily promoted approach for training ML models on sensitive data, e.g., text typed by users on their smartphones. FL is expressly designed for training on data that are unbalanced and non-iid across the participants. To ensure privacy and integrity of the fedeated model, latest FL approaches use differential privacy or robust aggregation. We look at FL from the \emph{local} viewpoint of an individual participant and ask: (1) do participants have an incentive to participate in FL? (2) how can participants \emph{individually} improve the quality of their local models, without re-designing the FL framework and/or involving other participants? First, we show that on standard tasks such as next-word prediction, many participants gain no benefit from FL because the federated model is less accurate on their data than the models they can train locally on their own. Second, we show that differential privacy and robust aggregation make this problem worse by further destroying the accuracy of the federated model for many participants. Then, we evaluate three techniques for local adaptation of federated models: fine-tuning, multi-task learning, and knowledge distillation. We analyze where each is applicable and demonstrate that all participants benefit from local adaptation. Participants whose local models are poor obtain big accuracy improvements over conventional FL. Participants whose local models are better than the federated model\textemdash and who have no incentive to participate in FL today\textemdash improve less, but sufficiently to make the adapted federated model better than their local models.
BibTeX
@misc{yu2020salvaging,
  author = {Tao Yu and Eugene Bagdasaryan and Vitaly Shmatikov},
  title = {{Salvaging Federated Learning by Local Adaptation}},
  year = {2020},
  eprint = {2002.04758},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.04758},
}
Figure