Abstract
Federated learning (FL) is designed for training on unbalanced and non-IID data across participants. However, from a local viewpoint, many participants lack the incentive to join because the federated model is often less accurate on their data than a locally trained model. This problem is exacerbated by differential privacy and robust aggregation. This paper evaluates local adaptation techniques—fine-tuning, multi-task learning, and knowledge distillation—demonstrating that they can restore the incentive to participate by ensuring the adapted federated model outperforms local-only models for nearly all users.
The Participation Incentive Problem
Federated Learning (FL) allows collaborative model training without sharing raw data, making it ideal for sensitive applications like next-word prediction on smartphones. However, a fundamental conflict exists: while FL aims to include diverse participants to improve robustness, the mechanisms used to protect privacy (Differential Privacy) and integrity (Robust Aggregation) often "average out" the unique contributions of individual users.
For many participants, especially those with high-quality or unique local data, the resulting global model is a "net loss"—it performs worse than a model they could have trained independently. If participation leads to a worse product, users have no rational incentive to join the federation.
The Impact of Privacy and Robustness
The authors demonstrate that standard privacy and robustness protections significantly degrade the model for many users. On the Reddit next-word prediction task, Differential Privacy (DP-FED) resulted in lower accuracy than local models for 21% of participants. Robust aggregation, designed to ignore outliers, was even more damaging, making the global model worse than local ones for 52% of users.
As illustrated in the scatter plots above, there is a clear negative correlation between local model quality and the benefit gained from FL. Users with the most unique data (the "head" of the distribution) are the ones most likely to be penalized by the global aggregation process, particularly under robust or private frameworks.
Salvaging FL through Local Adaptation
Rather than redesigning global aggregation, the authors propose that participants perform local adaptation on the received global model. They evaluated three primary methods:
- Fine-tuning (FT): Re-training the global model on local data.
- Multi-task Learning (MTL): Using techniques like Elastic Weight Consolidation (EWC) to learn local nuances without forgetting global patterns.
- Knowledge Distillation (KD): Using the global model as a "teacher" to guide a local "student" model, preventing overfitting on small local datasets.
Key Results
Local adaptation proved to be a universal win. In the Reddit task, it improved mean accuracy by over 2%, effectively recovering the performance lost to privacy and robustness mechanisms. In image classification (CIFAR-10), gains ranged from 3% to 7%.
Crucially, adaptation restored the incentive to participate. For the ROBUST-FED setup on Reddit, where tens of thousands of users originally had no reason to join, local adaptation made the federated model better than the local-only model for 99.96% of participants.
Conclusion
Local adaptation transforms FL from a system that potentially penalizes unique users into one that provides value to everyone. By allowing users to "tweak" the global model to their specific needs, we can maintain the privacy and robustness of the federation while ensuring every participant receives a superior model.
Summary updated