Introduction
Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across decentralized devices while maintaining data privacy. However, a significant challenge remains: data heterogeneity (non-IID data). When clients possess vastly different data distributions, a single global model often fails to provide optimal performance for individual participants.
Personalized Federated Learning (pFL) addresses this by tailoring models to specific clients. Traditional pFL methods typically determine the degree of collaboration between clients based on the similarity of their entire models. This paper argues that such "model-wise" aggregation is too coarse. By treating the model as a monolithic block, these methods ignore the functional diversity of neural network layers—where shallow layers often learn general features and deeper layers capture task-specific patterns. To solve this, the authors propose pFedLA (Layer-wised Personalized Federated Learning).
Methodology
The core innovation of pFedLA is the shift from model-level to layer-level aggregation. The framework introduces two primary components:
1. Layer-wise Hypernetworks
In pFedLA, the server maintains a dedicated, small-scale hypernetwork ($HN_i$) for each client. Each hypernetwork takes a learnable embedding vector ($v_i$)—representing the unique characteristics of a client's data—and outputs a set of aggregation weights. Unlike standard methods that assign one weight per client, pFedLA generates a distinct weight for every layer of every client. This allows the system to find fine-grained similarities; for instance, a client might share early-layer feature extractors with one group of peers while sharing late-layer classifiers with another.
2. HeurpFedLA (Heuristic Improvement)
To address communication overhead, the authors developed HeurpFedLA. This variant identifies "top-k" layers that have the highest self-contribution weights. These highly personalized layers are kept local and are neither uploaded to the server nor aggregated with other clients' updates. This heuristic not only saves bandwidth but also prevents "noise" from dissimilar clients from degrading the most critical parts of a personalized model.
Experimental Setup
The framework was evaluated across four standard image classification benchmarks: EMNIST, FashionMNIST, CIFAR-10, and CIFAR-100. To simulate real-world heterogeneity, the researchers used two non-IID scenarios:
- Pathological non-IID: Each client is restricted to only a few classes.
- Dirichlet non-IID: Clients have all classes but in highly unbalanced proportions.
Key Results and Insights
Superior Performance and Granularity
pFedLA consistently outperformed state-of-the-art pFL methods, including FedRep, pFedHN, and FedFomo. On CIFAR-10 with 100 clients, pFedLA achieved an accuracy of 73.15%, significantly higher than the 58.43% achieved by standard FedAvg.
As shown in the figures above, layer-wise aggregation converges faster and reaches higher accuracy than model-wise approaches. Furthermore, the weight distribution visualization confirms that the importance of different clients varies significantly across layers (Conv1 vs. FC3), justifying the need for granular aggregation.
The "Less is More" Communication Paradox
One of the most striking findings was that HeurpFedLA, despite communicating fewer parameters, often matched or exceeded the accuracy of the full pFedLA. By retaining the most "personal" layers locally, the model avoids the negative transfer of information from other clients. In CIFAR-10 tests, this approach reduced communication costs by approximately 40%.
Conclusion and Future Work
pFedLA demonstrates that the key to effective personalization in Federated Learning lies in the granularity of aggregation. By leveraging hypernetworks to learn layer-specific similarities, the framework handles extreme data heterogeneity more robustly than previous methods.
Future research directions include reducing the server-side memory overhead required to maintain multiple hypernetworks and developing automated methods for selecting the optimal number of local layers ($k$) in the heuristic variant to further streamline the training process.
Summary updated