FedPAE: Peer-Adaptive Ensemble Learning for Asynchronous and Model-Heterogeneous Federated Learning

Brianna Mueller, W. Nick Street, Stephen Baek, Qihang Lin, Jingyi Yang, Yankun Huang

Introduction

Standard Federated Learning (FL) typically relies on a centralized server to aggregate model updates from various clients. While effective in controlled environments, this architecture introduces several critical bottlenecks in real-world deployments: single points of failure, communication congestion at the server, and the assumption of model homogeneity. Furthermore, real-world data is rarely distributed uniformly (non-IID), leading to "statistical heterogeneity" where a single global model fails to perform well for all participants.

To address these challenges, we present FedPAE (Federated Peer-Adaptive Ensemble Learning), a fully decentralized framework designed to handle both statistical and systems heterogeneity. By moving away from centralized weight averaging and toward a peer-to-peer ensemble selection strategy, FedPAE allows diverse devices to collaborate without a middleman, effectively mitigating the risks of "negative transfer" from irrelevant peers.

The FedPAE Framework

FedPAE reimagines the FL workflow by replacing the central coordinator with a peer-to-peer sharing mechanism. The core innovation lies in how individual clients select and combine models from their peers to create a personalized ensemble.

1. Peer-to-Peer Sharing and the Model Bench

In FedPAE, clients train their models locally and share either their model weights or their predictions directly with peers. Each client maintains a "model bench"—a repository of candidate models. For resource-constrained devices, such as IoT sensors, FedPAE allows the storage of peer model predictions on a local validation set rather than the full model weights, significantly reducing memory overhead.

2. Multi-Objective Optimization via NSGA-II

Rather than simply selecting the most accurate peer models, FedPAE utilizes the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to optimize the ensemble. The algorithm seeks to balance two often-conflicting objectives:

  • Strength: Maximizing the individual accuracy of ensemble members on the client’s local data.
  • Diversity: Minimizing the correlation between model errors. High diversity ensures that different models in the ensemble fail on different types of inputs, which is the mathematical foundation for superior ensemble performance.

3. Asynchronous and Heterogeneous Learning

Because FedPAE does not rely on global synchronization rounds, it is inherently asynchronous. Clients can update their local ensembles whenever they receive new information from a peer. Furthermore, because the ensemble selection is based on model outputs rather than internal architectures, FedPAE naturally supports model heterogeneity—a powerful server and a low-power device can collaborate even if their neural network architectures are entirely different.

Experimental Evaluation

We evaluated FedPAE using two standard computer vision benchmarks: CIFAR-10 and CIFAR-100. To simulate the complexities of real-world data, we employed a Dirichlet distribution to partition the datasets, creating varying levels of data skew among clients.

Key Results

  • Superior Accuracy: In highly skewed CIFAR-10 environments, FedPAE achieved an accuracy of 87.3%. This significantly outperformed the standard FedAvg baseline (66.8%) and surpassed specialized personalized FL methods such as FedKD and FML (both at 87.0%).
  • Robustness to Heterogeneity: As the data became more non-IID, the performance gap between FedPAE and other methods widened. The framework’s "opt-out" mechanism allows clients to rely solely on local models if peer models are found to be harmful, effectively shielding the client from negative transfer.
  • Efficiency and Convergence: FedPAE demonstrated a remarkable reduction in total runtime. In some configurations, it reached peak performance 10x faster than competing personalized FL methods (e.g., 108 minutes compared to over 1,300 minutes for certain baselines), primarily because it requires fewer communication exchanges to achieve high accuracy.

Discussion and Future Directions

FedPAE demonstrates that diversity in federated learning is a feature to be leveraged rather than a problem to be solved. By embracing the differences between peer models, the framework creates robust, personalized ensembles that outperform global models.

However, challenges remain regarding communication volume, as transferring full models or large prediction matrices can be taxing on bandwidth. Future work will explore the use of clustering to help clients identify "kindred spirit" peers more efficiently, as well as dynamic ensemble selection, where the specific models used for inference are chosen on a per-image basis to further optimize performance and computational cost.

Summary updated

BibTeX
@inproceedings{mueller2024fedpae,
  author = {Brianna Mueller and W. Nick Street and Stephen Baek and Qihang Lin and Jingyi Yang and Yankun Huang},
  title = {{FedPAE: Peer-Adaptive Ensemble Learning for Asynchronous and Model-Heterogeneous Federated Learning}},
  booktitle = {IEEE Big Data},
  year = {2024},
  doi = {10.1109/bigdata62323.2024.10825304},
}
Figure