Layer-wised Model Aggregation for Personalized Federated Learning

Xiaosong Ma, Jie Zhang, Song Guo, Wenchao Xu

Introduction

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across decentralized devices while maintaining data privacy. However, a significant challenge remains: data heterogeneity (non-IID data). When clients possess vastly different data distributions, a single global model often fails to provide optimal performance for individual participants.

Personalized Federated Learning (pFL) addresses this by tailoring models to specific clients. Traditional pFL methods typically determine the degree of collaboration between clients based on the similarity of their entire models. This paper argues that such "model-wise" aggregation is too coarse. By treating the model as a monolithic block, these methods ignore the functional diversity of neural network layers—where shallow layers often learn general features and deeper layers capture task-specific patterns. To solve this, the authors propose pFedLA (Layer-wised Personalized Federated Learning).

Methodology

The core innovation of pFedLA is the shift from model-level to layer-level aggregation. The framework introduces two primary components:

1. Layer-wise Hypernetworks

In pFedLA, the server maintains a dedicated, small-scale hypernetwork ($HN_i$) for each client. Each hypernetwork takes a learnable embedding vector ($v_i$)—representing the unique characteristics of a client's data—and outputs a set of aggregation weights. Unlike standard methods that assign one weight per client, pFedLA generates a distinct weight for every layer of every client. This allows the system to find fine-grained similarities; for instance, a client might share early-layer feature extractors with one group of peers while sharing late-layer classifiers with another.

The pFedLA framework overview showing server-side hypernetworks generating personalized aggregation weights for each client.

2. HeurpFedLA (Heuristic Improvement)

To address communication overhead, the authors developed HeurpFedLA. This variant identifies "top-k" layers that have the highest self-contribution weights. These highly personalized layers are kept local and are neither uploaded to the server nor aggregated with other clients' updates. This heuristic not only saves bandwidth but also prevents "noise" from dissimilar clients from degrading the most critical parts of a personalized model.

Experimental Setup

The framework was evaluated across four standard image classification benchmarks: EMNIST, FashionMNIST, CIFAR-10, and CIFAR-100. To simulate real-world heterogeneity, the researchers used two non-IID scenarios:

  • Pathological non-IID: Each client is restricted to only a few classes.
  • Dirichlet non-IID: Clients have all classes but in highly unbalanced proportions.

Key Results and Insights

Superior Performance and Granularity

pFedLA consistently outperformed state-of-the-art pFL methods, including FedRep, pFedHN, and FedFomo. On CIFAR-10 with 100 clients, pFedLA achieved an accuracy of 73.15%, significantly higher than the 58.43% achieved by standard FedAvg.

(a) Accuracy comparison between layer-wise and model-wise aggregation. (b) Learned weights for different layers across multiple clients.

As shown in the figures above, layer-wise aggregation converges faster and reaches higher accuracy than model-wise approaches. Furthermore, the weight distribution visualization confirms that the importance of different clients varies significantly across layers (Conv1 vs. FC3), justifying the need for granular aggregation.

The "Less is More" Communication Paradox

One of the most striking findings was that HeurpFedLA, despite communicating fewer parameters, often matched or exceeded the accuracy of the full pFedLA. By retaining the most "personal" layers locally, the model avoids the negative transfer of information from other clients. In CIFAR-10 tests, this approach reduced communication costs by approximately 40%.

Conclusion and Future Work

pFedLA demonstrates that the key to effective personalization in Federated Learning lies in the granularity of aggregation. By leveraging hypernetworks to learn layer-specific similarities, the framework handles extreme data heterogeneity more robustly than previous methods.

Future research directions include reducing the server-side memory overhead required to maintain multiple hypernetworks and developing automated methods for selecting the optimal number of local layers ($k$) in the heuristic variant to further streamline the training process.

Summary updated

Abstract

Personalized Federated Learning (pFL) not only can capture the common priors from broad range of distributed data, but also support customized models for heterogeneous clients. Researches over the past few years have applied the weighted aggregation manner to produce personalized models, where the weights are determined by calibrating the distance of the entire model parameters or loss values, and have yet to consider the layer-level impacts to the aggregation process, leading to lagged model convergence and inadequate personalization over non-IID datasets. In this paper, we propose a novel pFL training framework dubbed Layer-wised Personalized Federated learning (pFedLA) that can discern the importance of each layer from different clients, and thus is able to optimize the personalized model aggregation for clients with heterogeneous data. Specifically, we employ a dedicated hypernetwork per client on the server side, which is trained to identify the mutual contribution factors at layer granularity. Meanwhile, a parameterized mechanism is introduced to update the layer-wised aggregation weights to progressively exploit the inter-user similarity and realize accurate model personalization. Extensive experiments are conducted over different models and learning tasks, and we show that the proposed methods achieve significantly higher performance than state-of-the-art pFL methods.
BibTeX
@inproceedings{ma2022layer,
  author = {Xiaosong Ma and Jie Zhang and Song Guo and Wenchao Xu},
  title = {{Layer-wised Model Aggregation for Personalized Federated Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00985},
}
Figure