Abstract
Federated learning (FL) enables collaborative model training across edge devices while preserving data privacy, but it is often constrained by communication bandwidth and device-side computational limits. This paper introduces a real-time, multi-objective evolutionary Neural Architecture Search (NAS) framework tailored for FL environments. By employing a novel "double-sampling" technique and weight sharing, the proposed method optimizes neural network architectures and computational costs simultaneously. This approach eliminates the massive communication overhead and retraining requirements of traditional NAS, making it highly practical for real-world deployment on mobile and edge devices.
Introduction
Traditional Neural Architecture Search (NAS) methods are computationally expensive, often requiring the training of hundreds of candidate models from scratch. In a Federated Learning (FL) context, where data is decentralized and communication is a primary bottleneck, "offline" evolutionary NAS is impractical. It demands excessive computational resources from edge devices and requires frequent transmission of large model parameters, which can lead to severe performance degradation due to random weight reinitialization.
To address these challenges, this research proposes a real-time federated evolutionary NAS. The goal is to perform architecture search during the standard FL training process, optimizing for both model accuracy and efficiency (e.g., FLOPs) without overwhelming the participating clients.
Methodology
The core of the proposed framework is a real-time federated evolutionary NAS utilizing the NSGA-II multi-objective optimization algorithm. The system achieves efficiency through two primary mechanisms: double-sampling and weight sharing.
The Master Model and Choice Blocks
Instead of searching an unconstrained space, the algorithm utilizes a predefined "master model." This model acts as a super-network containing various "choice blocks," such as identity, residual, inverted residual, and depthwise separable convolutions.
Double-Sampling Technique
The double-sampling approach operates at two levels:
- Model Sampling: The algorithm samples specific "sub-networks" from the master model. Each sub-network represents a unique architecture candidate in the evolutionary population.
- Client Sampling: Rather than broadcasting the entire master model, the server sends specific sub-networks to a randomly selected subset of clients for local training.
Weight Sharing and Aggregation
Crucially, sub-networks inherit their weights directly from the master model. This weight-sharing mechanism ensures that performance remains stable across generations and accelerates convergence. After local training, the server aggregates these sub-networks by mapping updated weights back into the master model structure. This allows an entire generation of evolutionary search to be completed within a single standard FL communication round.
Experimental Setup
The framework was evaluated using the CIFAR-10 dataset in both IID (Independent and Identically Distributed) and non-IID federated environments. To simulate real-world data heterogeneity in the non-IID setup, each client was restricted to images from only 5 out of the 10 classes.
Key Results and Discussion
The experimental results demonstrate significant improvements over standard baselines and traditional NAS methods:
- Superior Accuracy and Efficiency: In a 10-client IID setup, the search identified a model achieving 86.68% accuracy with only 0.2796G FLOPs. This significantly outperformed a ResNet18 baseline, which achieved 81.14% accuracy with nearly double the computational cost (0.5587G FLOPs).
- Optimal Trade-offs: The algorithm successfully found a "knee" solution—the optimal balance between performance and cost—achieving 83.36% accuracy with only 0.031G FLOPs, representing an 18x reduction in computation compared to ResNet18.
- Operational Speed: The real-time approach is approximately 5 times faster than offline evolutionary NAS per search round. By embedding the search into the training process, it avoids the lengthy retraining phases required by traditional methods.
- Impact of Data Distribution: While highly effective, performance showed slight degradation in non-IID settings or when the client pool increased to 50, as the reduced data per client can make local updates less representative of the global distribution.
Conclusion and Future Work
This paper presents a viable path for implementing NAS in resource-constrained federated environments. By leveraging weight sharing and double-sampling, the framework enables real-time architecture evolution that is both communication-efficient and computationally feasible for edge devices.
Future research will focus on scaling the algorithm to thousands of clients and adapting the framework for vertically partitioned data, where different clients hold different features for the same set of users. Additionally, addressing the challenges of batch normalization in non-IID settings remains a priority to further enhance the absolute performance of the discovered models.
Summary updated