Hypernetwork Aggregation for Decentralized Personalized Federated Learning

Weishi Li, Yong Peng, Mengyao Du, Fuhui Sun, Xiaoyan Wang, Li ShenIJCAI 2025 · DOI

Abstract

Decentralized Federated Learning (DFL) eliminates the need for a central server, mitigating single points of failure but introducing significant communication overhead and challenges in handling non-IID data. We introduce DFedHP, a novel framework that utilizes hypernetworks as a communication proxy. By exchanging small hypernetwork parameters instead of massive model weights, DFedHP reduces communication costs by up to 87% while enhancing model personalization and convergence speed in highly heterogeneous environments.

Introduction

Standard Federated Learning (FL) typically relies on a central server for model aggregation, which can become a bottleneck and a privacy risk. Decentralized FL (DFL) allows peer-to-peer communication but often requires nodes to exchange full model parameters, consuming excessive bandwidth—a major hurdle for mobile or IoT devices. Furthermore, real-world data is typically non-IID (Independent and Identically Distributed), necessitating Personalized FL (PFL) to balance global knowledge with local expertise. DFedHP addresses these challenges by decoupling communication from model size through the use of hypernetworks.

Methodology: Decentralized Hypernetwork Aggregation

The core of DFedHP is the use of hypernetworks to generate the shared weights of a local model. Instead of direct weight sharing, clients maintain and communicate a small hypernetwork ($\phi$) and an embedding vector ($z$).

Overview of the DFedHP framework showing decentralized hypernetwork aggregation and local weight generation.

The process involves several key components:

  1. Weight Generation: A hypernetwork $h(z, \phi)$ takes an embedding vector as input and outputs the weights for the shared portion of the local model (e.g., convolutional layers).
  2. Model Decoupling: The local model is split into a shared part (generated by the hypernetwork) and a personalized part (e.g., the classification head) that is trained locally and never shared.
  3. Lean Communication: During aggregation, clients exchange only the hypernetwork parameters $\phi$ and embeddings $z$. Since these are significantly smaller than the target model, bandwidth usage is drastically reduced.
  4. Decentralized Aggregation: Clients average the received hypernetwork parameters based on the network topology (e.g., ring or grid), allowing global knowledge to diffuse through the network.

Experimental Evaluation

The authors evaluated DFedHP on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. To simulate realistic heterogeneity, data was partitioned using Dirichlet and Pathological distributions. DFedHP was compared against state-of-the-art baselines including DFedAvg, DisPFL, and Ditto.

Key Results and Discussion

DFedHP demonstrates significant improvements in both efficiency and performance across various settings.

Test accuracy vs. communication rounds across CIFAR-10, CIFAR-100, and Tiny-ImageNet under various non-IID settings.
  • Massive Bandwidth Savings: DFedHP achieves up to an 87.81% reduction in the number of parameters transmitted per round compared to standard DFL methods.
  • Superior Accuracy and Convergence: In highly heterogeneous settings (e.g., CIFAR-10 Dirichlet 0.3), DFedHP reached 77.01% accuracy, surpassing DFedAvg (73.96%). It also converged up to 39.29% faster in complex scenarios like CIFAR-100 Pathological 20.
  • Privacy and Robustness: Sharing hypernetwork parameters adds a layer of abstraction, making it harder for adversaries to perform gradient inversion attacks. Additionally, the framework is "plug-and-play," enhancing other DFL algorithms when integrated as a performance-boosting plugin.

Conclusion

DFedHP offers a scalable and efficient solution for decentralized personalized federated learning. By leveraging hypernetworks, it breaks the linear relationship between model size and communication cost. Future work will explore more complex hypernetwork architectures for large-scale models like Transformers and investigate the compute-communication trade-off on resource-constrained hardware.

Summary updated

BibTeX
@inproceedings{li2025hypernetwork,
  author = {Weishi Li and Yong Peng and Mengyao Du and Fuhui Sun and Xiaoyan Wang and Li Shen},
  title = {{Hypernetwork Aggregation for Decentralized Personalized Federated Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/161},
}
Figure