FedAS: Bridging Inconsistency in Personalized Federated Learning

Xiyuan Yang, Wenke Huang, Mang Ye

Introduction

Federated Learning (FL) enables collaborative model training across distributed parties without centralizing private data. In real-world scenarios, data heterogeneity (non-IID data) necessitates Personalized Federated Learning (PFL), which typically decouples a model into a shared global backbone and a personalized local head. However, current PFL paradigms suffer from two critical "inconsistency" gaps that degrade performance.

The Inconsistency Problem

The authors identify two primary challenges in existing PFL methods:

  1. Intra-client Inconsistency: When a client receives an updated global backbone, it often conflicts with the existing local head. This mismatch between the two model components leads to suboptimal feature representation.
  2. Inter-client Inconsistency: Real-world FL environments involve "stragglers"—clients with limited connectivity or compute power. These under-trained clients contribute noisy or incomplete updates that can distort the global model during aggregation.
Figure 2: Illustration of Intra-client and Inter-client inconsistency in personalized federated learning.

The FedAS Framework

To address these issues, the authors propose FedAS (Federated Parameter-Alignment and Client-Synchronization), a dual-step framework designed to harmonize model updates and filter out noise from stragglers.

Federated Parameter-Alignment (PA)

FedAS introduces a "localization" step before local training begins. Instead of immediately training on the new global backbone, the client aligns the feature outputs of the new backbone with those of the previous local version. This "self-alignment" ensures that the global parameters are adapted to the local head's specific feature space. Crucially, this alignment happens serially before local optimization, providing a consistent foundation for training rather than attempting to fix mismatches during the training process.

Client-Synchronization (CS)

To mitigate the impact of stragglers, FedAS employs a weighted aggregation scheme based on the trace of the Fisher Information Matrix (t-FIM). t-FIM serves as a proxy for the "information content" or "sharpness" of a client's update. Well-trained, active clients exhibit higher t-FIM values, while under-trained stragglers have lower values. By weighting updates according to t-FIM, the server effectively prioritizes high-quality knowledge and filters out the noise introduced by stragglers, all without requiring access to raw client data.

Figure 5: The FedAS workflow, detailing the Parameter Alignment and Client Synchronization stages across active and straggler clients.

Experimental Evaluation

FedAS was evaluated on CIFAR-10 and CIFAR-100 datasets under varying degrees of data heterogeneity (using Dirichlet distribution $\beta$) and client availability (online ratios $P$).

Key Results

  • State-of-the-Art Performance: FedAS consistently outperformed leading PFL methods like FedRep, FedPAC, and FedALA. On CIFAR-100 with high heterogeneity ($\beta=0.1$) and low client availability ($P=0.2$), FedAS achieved 52.41% accuracy, outperforming the next best method (FedBABU) by 2.56%.
  • Resilience to Stragglers: Unlike standard methods that degrade sharply as client participation drops, FedAS maintains stable performance. Even when only 20% of clients were active, it maintained a lead of 4-6% over most competitors.
  • Fast Convergence: Training curves show that FedAS reaches higher accuracy levels faster than standard approaches, as the alignment step reduces the parameter conflicts that typically hinder optimization.

Discussion and Future Work

While FedAS effectively bridges inconsistency gaps, it currently assumes a homogeneous model architecture across all clients. Future research could extend these alignment techniques to model-heterogeneous environments where devices may require different model sizes. Additionally, while using the diagonal of the Fisher Information Matrix reduces memory overhead, the computational cost of gradient calculations remains a consideration for resource-constrained edge devices.

Summary updated

Abstract

Personalized Federated Learning (PFL) is primarily de-signed to provide customized models for each client to better fit the non-iid distributed client data, which is a inherent challenge in Federated Learning. However, current PFL methods suffer from inconsistencies in both intra-client and inter-client levels: 1) The intra-client inconsistency stems from the asynchronous update strategy for personalized and shared parameters. In PFL, clients update their shared parameters to communicate and learn from others, while keeping personalized parts unchanged, leading to poor coordination between these two components. 2) The Inter-client inconsistency arises from “stragglers” - inactive clients that communicate and train with the server less frequently. This results in their under-trained personalized models and impedes the collaborative training stage for other clients. In this paper, we present a novel PFL framework named FedAS, which uses Federated Parameter-Alignment and Client-Synchronization to overcome above challenges. Initially, we enhance the localization of global parameters by infusing them with local insights. We make the shared parts learn from previous model, thereby increasing their local relevance and reducing the impact of parameter inconsistency. Furthermore, we design a robust aggregation method to mitigate the impact of stragglers by preventing the incorporation of their under-trained knowledge into aggregated model. Experimental results on CifarlO and CifarlOO validate the effectiveness of our FedAS in achieving better performance and robustness against data heterogeneity.
BibTeX
@inproceedings{yang2024fedas,
  author = {Xiyuan Yang and Wenke Huang and Mang Ye},
  title = {{FedAS: Bridging Inconsistency in Personalized Federated Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01139},
}
Figure