Real-Time Federated Evolutionary Neural Architecture Search

Hangyu Zhu, Yaochu Jin

Abstract

Federated learning (FL) enables collaborative model training across edge devices while preserving data privacy, but it is often constrained by communication bandwidth and device-side computational limits. This paper introduces a real-time, multi-objective evolutionary Neural Architecture Search (NAS) framework tailored for FL environments. By employing a novel "double-sampling" technique and weight sharing, the proposed method optimizes neural network architectures and computational costs simultaneously. This approach eliminates the massive communication overhead and retraining requirements of traditional NAS, making it highly practical for real-world deployment on mobile and edge devices.

Introduction

Traditional Neural Architecture Search (NAS) methods are computationally expensive, often requiring the training of hundreds of candidate models from scratch. In a Federated Learning (FL) context, where data is decentralized and communication is a primary bottleneck, "offline" evolutionary NAS is impractical. It demands excessive computational resources from edge devices and requires frequent transmission of large model parameters, which can lead to severe performance degradation due to random weight reinitialization.

To address these challenges, this research proposes a real-time federated evolutionary NAS. The goal is to perform architecture search during the standard FL training process, optimizing for both model accuracy and efficiency (e.g., FLOPs) without overwhelming the participating clients.

Methodology

The core of the proposed framework is a real-time federated evolutionary NAS utilizing the NSGA-II multi-objective optimization algorithm. The system achieves efficiency through two primary mechanisms: double-sampling and weight sharing.

The Master Model and Choice Blocks

Instead of searching an unconstrained space, the algorithm utilizes a predefined "master model." This model acts as a super-network containing various "choice blocks," such as identity, residual, inverted residual, and depthwise separable convolutions.

The macro-architecture of the master model and the internal structure of a Choice Block, which allows for flexible sub-network sampling.

Double-Sampling Technique

The double-sampling approach operates at two levels:

  1. Model Sampling: The algorithm samples specific "sub-networks" from the master model. Each sub-network represents a unique architecture candidate in the evolutionary population.
  2. Client Sampling: Rather than broadcasting the entire master model, the server sends specific sub-networks to a randomly selected subset of clients for local training.

Weight Sharing and Aggregation

Crucially, sub-networks inherit their weights directly from the master model. This weight-sharing mechanism ensures that performance remains stable across generations and accelerates convergence. After local training, the server aggregates these sub-networks by mapping updated weights back into the master model structure. This allows an entire generation of evolutionary search to be completed within a single standard FL communication round.

The real-time federated evolutionary NAS workflow, illustrating how sub-networks are downloaded by clients, updated locally, and uploaded for server-side aggregation.

Experimental Setup

The framework was evaluated using the CIFAR-10 dataset in both IID (Independent and Identically Distributed) and non-IID federated environments. To simulate real-world data heterogeneity in the non-IID setup, each client was restricted to images from only 5 out of the 10 classes.

Key Results and Discussion

The experimental results demonstrate significant improvements over standard baselines and traditional NAS methods:

  • Superior Accuracy and Efficiency: In a 10-client IID setup, the search identified a model achieving 86.68% accuracy with only 0.2796G FLOPs. This significantly outperformed a ResNet18 baseline, which achieved 81.14% accuracy with nearly double the computational cost (0.5587G FLOPs).
  • Optimal Trade-offs: The algorithm successfully found a "knee" solution—the optimal balance between performance and cost—achieving 83.36% accuracy with only 0.031G FLOPs, representing an 18x reduction in computation compared to ResNet18.
  • Operational Speed: The real-time approach is approximately 5 times faster than offline evolutionary NAS per search round. By embedding the search into the training process, it avoids the lengthy retraining phases required by traditional methods.
  • Impact of Data Distribution: While highly effective, performance showed slight degradation in non-IID settings or when the client pool increased to 50, as the reduced data per client can make local updates less representative of the global distribution.

Conclusion and Future Work

This paper presents a viable path for implementing NAS in resource-constrained federated environments. By leveraging weight sharing and double-sampling, the framework enables real-time architecture evolution that is both communication-efficient and computationally feasible for edge devices.

Future research will focus on scaling the algorithm to thousands of clients and adapting the framework for vertically partitioned data, where different clients hold different features for the same set of users. Additionally, addressing the challenges of batch normalization in non-IID settings remains a priority to further enhance the absolute performance of the discovered models.

Summary updated

Abstract

Federated learning is a distributed machine learning approach to privacy preservation and two major technical challenges prevent a wider application of federated learning. One is that federated learning raises high demands on communication resources, since a large number of model parameters must be transmitted between the server and clients. The other challenge is that training large machine learning models such as deep neural networks in federated learning requires a large amount of computational resources, which may be unrealistic for edge devices such as mobile phones. The problem becomes worse when deep neural architecture search (NAS) is to be carried out in federated learning. To address the above challenges, we propose an evolutionary approach to real-time federated NAS that not only optimizes the model performance but also reduces the local payload. During the search, a double-sampling technique is introduced, in which for each individual, only a randomly sampled submodel is transmitted to a number of randomly sampled clients for training. This way, we effectively reduce computational and communication costs required for evolutionary optimization, making the proposed framework well suitable for real-time federated NAS.
BibTeX
@article{zhu2022real,
  author = {Hangyu Zhu and Yaochu Jin},
  title = {{Real-Time Federated Evolutionary Neural Architecture Search}},
  journal = {IEEE Trans. Evol. Comput.},
  year = {2022},
  doi = {10.1109/tevc.2021.3099448},
}
Figure