How to Prevent the Poor Performance Clients for Personalized Federated Learning?

Zhe Qu, Xingyu Li, Xiao Han, Rui Duan, Chengchao Shen, Lixing ChenCVPR 2023 · OpenReview

Abstract

Personalized federated learning (pFL) collaboratively trains personalized models, which provides a customized model solution for individual clients in the presence of heterogeneous distributed local data. Although many recent studies have applied various algorithms to enhance personalization in pFL, they mainly focus on improving the performance from averaging or top perspective. However, part of the clients may fall into poor performance and are not clearly discussed. Therefore, how to prevent these poor clients should be considered critically. Intuitively, these poor clients may come from biased universal information shared with others. To address this issue, we propose a novel pFL strategy, called Personalize Locally, Generalize Universally (PLGU). PLGU generalizes the fine-grained universal information and moderates its biased performance by designing a Layer-Wised Sharpness Aware Minimization (LWSAM) algorithm while keeping the personalization local. Specifically, we embed our proposed PLGU strategy into two pFL schemes concluded in this paper: with/without a global model, and present the training procedures in detail. Through in-depth study, we show that the proposed PLGU strategy achieves competitive generalization bounds on both considered pFL schemes. Our extensive experimental results show that all the proposed PLGU based-algorithms achieve state-of-the-art performance.

Introduction

Federated Learning (FL) allows multiple participants to train a model together without sharing their raw data. However, because client data is often non-IID (meaning every client’s data looks different), a one-size-fits-all global model rarely works. Personalized Federated Learning (pFL) solves this by giving each client a customized version of the model.

A critical oversight in current pFL research is that most algorithms focus on improving average performance or top performers. In practice, a significant percentage of clients may still end up with poor performance models. As shown in Figure 1, existing methods like pFedMe and FedRep leave a "tail" of clients with frustratingly low accuracy. In high-stakes fields like medicine, a model that fails for 10% of hospitals could lead to serious malpractice. These "poor clients" suffer because the shared universal information is biased toward the majority, making it a poor fit for their specific local data.

Distribution of client performance for pFedMe and FedRep, highlighting the "poor clients" tail.

The PLGU Strategy

To address this, we propose PLGU (Personalize Locally, Generalize Universally). It is built on two main pillars:

  1. Layer-Wised Sharpness-Aware Minimization (LWSAM): Standard SAM seeks "flat" minima in the loss landscape for better generalization. PLGU refines this by applying SAM at a layer-wise level, calculating a "personalization score" for each layer to determine its contribution to local features versus global concepts.
  2. Two-Scheme Integration:
    • Scheme I (With a Global Model): We propose PLGU-LF (Layer Freezing), which identifies and freezes personalization-critical layers locally while using LWSAM on universal layers.
  3. Scheme II (Without a Global Model): We apply the strategy to representation-based learning, creating PLGU-GRep, ensuring the shared feature extractor is smoothed and generalized.

Experimental Evaluation

We evaluated PLGU using three standard benchmarks: CIFAR10 (100 clients, 3 classes each), CIFAR100 (100 clients, 10 classes each), and Tiny-ImageNet (20 clients, 30 classes each).

Accuracy vs. communication rounds for CIFAR10, CIFAR100, and TmgNet across various pFL methods.

As shown in Figure 4, PLGU-based algorithms consistently outperformed state-of-the-art benchmarks like FedAvg, pFedMe, and FedRep. On CIFAR100, PLGU-LF achieved a 56.85% accuracy for the bottom 5% of clients, an increase of over 2% compared to other leading methods. Visualizations of loss landscapes confirmed that PLGU creates a much smoother surface for the global model, directly correlating to improved generalization.

Discussion and Conclusion

The "aha moment" of this work is the realization that poor local performance often stems from biased universal information. By using LWSAM to "smooth out" shared parts of the model, PLGU makes global knowledge more flexible for clients with unusual data distributions.

While PLGU-based methods achieve state-of-the-art performance, they do come with a computational trade-off. SAM-based optimizers typically require two forward/backward passes per update, doubling local computation. Future work could explore combining PLGU with communication-efficient techniques like quantization to further optimize the strategy for resource-constrained environments.

Summary updated

Abstract

Personalized federated learning (pFL) collaboratively trains personalized models, which provides a customized model solution for individual clients in the presence of heterogeneous distributed local data. Although many recent studies have applied various algorithms to enhance personalization in pFL, they mainly focus on improving the performance from averaging or top perspective. However, part of the clients may fall into poor performance and are not clearly discussed. Therefore, how to prevent these poor clients should be considered critically. Intuitively, these poor clients may come from biased universal information shared with others. To address this issue, we propose a novel pFL strategy, called Personalize Locally, Generalize Universally (PLGU). PLGU generalizes the fine-grained universal information and moderates its biased performance by designing a Layer-Wised Sharpness Aware Minimization (LWSAM) algorithm while keeping the personalization local. Specifically, we embed our proposed PLGU strategy into two pFL schemes concluded in this paper: with/without a global model, and present the training procedures in detail. Through in-depth study, we show that the proposed PLGU strategy achieves competitive generalization bounds on both considered pFL schemes. Our extensive experimental results show that all the proposed PLGU based-algorithms achieve state-of-the-art performance.
BibTeX
@inproceedings{qu2023how,
  author = {Zhe Qu and Xingyu Li and Xiao Han and Rui Duan and Chengchao Shen and Lixing Chen},
  title = {{How to Prevent the Poor Performance Clients for Personalized Federated Learning?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=4wP34iK7j7},
}
Figure