Personalized Federated Learning towards Communication Efficiency, Robustness and Fairness

Shiyun Lin, Yuze Han, Xiang Li, Zhihua ZhangNeurIPS 2022 · OpenReview

TL;DR

Researchers have introduced lp-proj, a new Personalized Federated Learning (PFL) framework that simultaneously tackles three of the biggest hurdles in the field: high communication costs, vulnerability to adversarial attacks, and performance unfairness. By projecting local models into a shared, fixed, low-dimensional random subspace, the method achieves massive communication savings (up to 1320x) while remaining resilient to malicious clients.

The Challenge in Federated Learning

Federated Learning (FL) allows multiple devices (clients) to train a model collaboratively without sharing their raw data. However, three major "pain points" often hinder its real-world application:

  1. Communication Efficiency: Moving large model updates across the network is slow and expensive.
  2. Robustness: FL systems are decentralized, making them easy targets for attacks where malicious clients send corrupted updates to sabotage the global model.
  3. Fairness: Because data is heterogeneous (non-IID) across devices, a single global model often performs well for some clients but poorly for others.

While existing "Personalized" FL methods try to address these by letting clients deviate slightly from the global model, most focus on just one or two of these issues. This paper asks: Can we balance all three constraints at once?

The lp-proj Framework

The authors propose lp-proj (Projection-based $L^p$ Regularized PFL). Its core innovation lies in how it bridges the gap between a global "reference" model and local "personalized" models:

  • Low-Dimensional Random Projection: Instead of communicating the entire high-dimensional model, the server and clients agree on a fixed, random, low-dimensional subspace at the start of training. Only the components of the model within this subspace are communicated.
  • Infimal Convolution: This mathematical technique acts as a flexible regularizer. It allows local models to adapt to their specific data while staying anchored to the global model within the shared subspace.
  • $L^p$-Regularization: By choosing different values for $p$ (specifically $p=1$ or $p=2$), the method can tune its properties. For instance, $L^1$ regularization is known to be more robust against outliers and noise, which helps defend against adversarial attacks.

Key Insights

  • The Subspace as a Filter: Random projection isn't just a compression tool; it acts as a natural defense mechanism. By forcing communication through a low-dimensional "bottleneck," many types of adversarial noise are filtered out before they reach the server.
  • Personalization via "Beyond-Subspace" Components: Local models are allowed to have components outside the shared subspace. This allows them to capture the unique features of local data (personalization) while the shared subspace captures general knowledge.
  • Fairness through Consensus: By requiring "near consensus" only in the low-dimensional subspace, the system provides enough flexibility for all clients to reach a high level of accuracy, leading to a more uniform performance distribution across the network.

Key Results

The authors evaluated their method on standard machine learning benchmarks (EMNIST, CIFAR-10) and synthetic datasets designed to test data heterogeneity and adversarial intensity.

1. Communication Efficiency

In experiments on EMNIST and Synthetic data, lp-proj saved between 79x and 1320x in communication costs compared to the best competing methods while reaching the same target accuracy.

2. Robustness Against Attacks

Under attacks where malicious clients corrupt their updates, lp-proj remained highly stable. Even when 20% of the clients were malicious, lp-proj maintained its performance, while most other state-of-the-art methods (like Ditto or pFedMe) experienced significant performance drops.

Test Accuracy vs. Fraction of Malicious Clients

3. Fairness Across Clients

The method achieved a lower variance in test accuracy across clients, meaning no single client was left behind. On CIFAR-10, lp-proj-1 achieved the highest overall test accuracy (79.22%) while simultaneously recording the lowest variance (0.0097) among all competitors.

Variance vs. Test Accuracy on CIFAR-10

4. Convergence

The authors provided theoretical proof that the method converges for smooth objectives. Crucially, the dependence on the projection dimension is "mild," meaning you don't need a huge subspace to get good results.

Limitations and Future Work

While the results are promising, the authors note a few areas for further exploration:

  • Theoretical Scope: The current convergence theory primarily covers the $L^2$ case and smooth objectives. Extending this to the $L^1$ case and more complex non-convex models (like deep neural networks) is a logical next step.
  • Privacy: While FL is inherently more private than centralized training, the paper does not explicitly incorporate formal privacy guarantees like Differential Privacy, which could be an interesting constraint to add to their framework.
  • Subspace Choice: The subspace is currently fixed and random. Future work could investigate whether "learning" the subspace or adapting it over time could yield even better performance.

Summary updated

Abstract

Personalized Federated Learning faces many challenges such as expensive communication costs, training-time adversarial attacks, and performance unfairness across devices. Recent developments witness a trade-off between a reference model and local models to achieve personalization. We follow the avenue and propose a personalized FL method towards the three goals. When it is time to communicate, our method projects local models into a shared-and-fixed low-dimensional random subspace and uses infimal convolution to control the deviation between the reference model and projected local models. We theoretically show our method converges for smooth objectives with square regularizers and the convergence dependence on the projection dimension is mild. We also illustrate the benefits of robustness and fairness on a class of linear problems. Finally, we conduct a large number of experiments to show the empirical superiority of our method over several state-of-the-art methods on the three aspects.
BibTeX
@inproceedings{lin2022personalized,
  author = {Shiyun Lin and Yuze Han and Xiang Li and Zhihua Zhang},
  title = {{Personalized Federated Learning towards Communication Efficiency, Robustness and Fairness}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=wFymjzZEEkH},
}
Figure