Abstract
Personalized Federated Learning (pFL) addresses the challenges of data heterogeneity (non-IID data) by developing customized models for individual clients. This paper introduces a "knowledge-aware parameter coaching" scheme that leverages a novel Relation Cube to capture layer-wise similarities between clients. By allowing clients to selectively "borrow" knowledge from specific layers of other clients' models, the proposed method achieves state-of-the-art accuracy and faster convergence without the heavy computational overhead associated with hypernetwork-based approaches.
Introduction
Standard Federated Learning (FL) aims to train a single global model across distributed clients. However, real-world data is often non-IID, leading to significant performance degradation for a unified model. Personalized FL (pFL) addresses this by providing each client with a custom model. Current pFL methods often suffer from two main drawbacks: coarse-grained aggregation that treats the entire model as a single block, and high computational overhead from complex architectures like hypernetworks.
Methodology: Knowledge-Aware Parameter Coaching
The proposed framework shifts the server's role from a simple aggregator to a "coach."
The Relation Cube ($R$)
The core innovation is the Relation Cube, a 3D structure ($N \times L \times N$) that explicitly stores the similarity between client $i$ and client $j$ at each neural network layer $l$. This allows for fine-grained knowledge sharing, recognizing that shallow layers (capturing general features like edges) may be more similar across clients than deeper, task-specific layers.
Parameter Coaching and Regularization
The server maintains a unique "coaching model" for every client, constructed as a weighted combination of other clients' parameters based on the Relation Cube. During local training, clients optimize their local loss while remaining close to their personalized coaching model through a novel regularization term. This process is optimized using an alternating method where the server updates the Relation Cube and clients update local parameters iteratively.
Experimental Results
The method was evaluated on standard benchmarks including MNIST, FMNIST, CIFAR-10, and CIFAR-100, with varying levels of data heterogeneity simulated via a Dirichlet distribution ($\alpha = 0.1, 0.3$).
Accuracy and Convergence
The proposed method consistently outperforms baselines such as FedAvg, FedProx, and FedAMP. On the CIFAR-100 dataset with high heterogeneity ($\alpha = 0.1$), it achieved a peak accuracy of 45.49%, significantly surpassing the next best baseline.
As shown in the figure above, the "coaching" method demonstrates superior convergence speed compared to other layer-wise methods like pFedLA. It reaches higher accuracy levels in fewer communication rounds, proving its efficiency across different datasets and levels of non-IID data.
Conclusion and Future Work
The knowledge-aware parameter coaching scheme effectively balances personalization and knowledge sharing through the Relation Cube. While achieving state-of-the-art results, future work will address the scalability of the Relation Cube for millions of devices and investigate potential privacy implications of the similarity mapping.
Summary updated