Knowledge-Aware Parameter Coaching for Personalized Federated Learning

Mingjian Zhi, Yuanguo Bi, Wenchao Xu, Haozhao Wang, Tianao Xiang

Abstract

Personalized Federated Learning (pFL) addresses the challenges of data heterogeneity (non-IID data) by developing customized models for individual clients. This paper introduces a "knowledge-aware parameter coaching" scheme that leverages a novel Relation Cube to capture layer-wise similarities between clients. By allowing clients to selectively "borrow" knowledge from specific layers of other clients' models, the proposed method achieves state-of-the-art accuracy and faster convergence without the heavy computational overhead associated with hypernetwork-based approaches.

Introduction

Standard Federated Learning (FL) aims to train a single global model across distributed clients. However, real-world data is often non-IID, leading to significant performance degradation for a unified model. Personalized FL (pFL) addresses this by providing each client with a custom model. Current pFL methods often suffer from two main drawbacks: coarse-grained aggregation that treats the entire model as a single block, and high computational overhead from complex architectures like hypernetworks.

Methodology: Knowledge-Aware Parameter Coaching

The proposed framework shifts the server's role from a simple aggregator to a "coach."

The Relation Cube ($R$)

The core innovation is the Relation Cube, a 3D structure ($N \times L \times N$) that explicitly stores the similarity between client $i$ and client $j$ at each neural network layer $l$. This allows for fine-grained knowledge sharing, recognizing that shallow layers (capturing general features like edges) may be more similar across clients than deeper, task-specific layers.

Parameter Coaching and Regularization

The server maintains a unique "coaching model" for every client, constructed as a weighted combination of other clients' parameters based on the Relation Cube. During local training, clients optimize their local loss while remaining close to their personalized coaching model through a novel regularization term. This process is optimized using an alternating method where the server updates the Relation Cube and clients update local parameters iteratively.

Experimental Results

The method was evaluated on standard benchmarks including MNIST, FMNIST, CIFAR-10, and CIFAR-100, with varying levels of data heterogeneity simulated via a Dirichlet distribution ($\alpha = 0.1, 0.3$).

Accuracy and Convergence

The proposed method consistently outperforms baselines such as FedAvg, FedProx, and FedAMP. On the CIFAR-100 dataset with high heterogeneity ($\alpha = 0.1$), it achieved a peak accuracy of 45.49%, significantly surpassing the next best baseline.

Test accuracy vs. communication rounds on CIFAR-10 and CIFAR-100 datasets under different heterogeneity levels ($\alpha$).

As shown in the figure above, the "coaching" method demonstrates superior convergence speed compared to other layer-wise methods like pFedLA. It reaches higher accuracy levels in fewer communication rounds, proving its efficiency across different datasets and levels of non-IID data.

Conclusion and Future Work

The knowledge-aware parameter coaching scheme effectively balances personalization and knowledge sharing through the Relation Cube. While achieving state-of-the-art results, future work will address the scalability of the Relation Cube for millions of devices and investigate potential privacy implications of the similarity mapping.

Summary updated

Abstract

Personalized Federated Learning (pFL) can effectively exploit the non-IID data from distributed clients by customizing personalized models. Existing pFL methods either simply take the local model as a whole for aggregation or require significant training overhead to induce the inter-client personalized weights, and thus clients cannot efficiently exploit the mutually relevant knowledge from each other. In this paper, we propose a knowledge-aware parameter coaching scheme where each client can swiftly and granularly refer to parameters of other clients to guide the local training, whereby accurate personalized client models can be efficiently produced without contradictory knowledge. Specifically, a novel regularizer is designed to conduct layer-wise parameters coaching via a relation cube, which is constructed based on the knowledge represented by the layered parameters among all clients. Then, we develop an optimization method to update the relation cube and the parameters of each client. It is theoretically demonstrated that the convergence of the proposed method can be guaranteed under both convex and non-convex settings. Extensive experiments are conducted over various datasets, which show that the proposed method can achieve better performance compared with the state-of-the-art baselines in terms of accuracy and convergence speed.
BibTeX
@inproceedings{zhi2024knowledge,
  author = {Mingjian Zhi and Yuanguo Bi and Wenchao Xu and Haozhao Wang and Tianao Xiang},
  title = {{Knowledge-Aware Parameter Coaching for Personalized Federated Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i15.29651},
}
Figure