Abstract
To preserve user privacy while enabling mobile intelligence, techniques have been proposed to train deep neural networks on decentralized data. However, training over decentralized data makes the design of neural architecture quite difficult as it already was. Such difficulty is further amplified when designing and deploying different neural architectures for heterogeneous mobile platforms. In this work, we propose an automatic neural architecture search into the decentralized training, as a new DNN training paradigm called Federated Neural Architecture Search, namely FedNAS. To deal with the primary challenge of limited on-client computational and communication resources, we present FedNAS, a highly optimized framework for efficient federated NAS. FedNAS fully exploits the key opportunity of insufficient model candidate re-training during the architecture search process, and incorporates three key optimizations: parallel candidates training on partial clients, early dropping candidates with inferior performance, and dynamic round numbers. Tested on large-scale datasets and typical CNN architectures, FedNAS achieves comparable model accuracy as state-of-the-art NAS algorithm that trains models with centralized data, and also reduces the client cost by up to two orders of magnitude compared to a straightforward design of federated NAS.
Introduction
Deep learning currently faces two major hurdles: the "design problem" and the "data problem." Finding the optimal neural network architecture (Neural Architecture Search, or NAS) is traditionally a labor-intensive process requiring thousands of GPU hours and access to massive, centralized datasets. Simultaneously, user data is increasingly private, leading to the rise of Federated Learning (FL), which allows models to be trained across thousands of devices without data ever leaving the device.
The challenge lies in the fact that NAS and FL are fundamentally at odds. NAS requires exploring hundreds of model candidates, a process that is often prohibitive for the limited battery and bandwidth of mobile devices. FedNAS (also referred to as DecNAS) addresses this by making automated model design feasible for resource-constrained environments.
The FedNAS Framework
FedNAS is built on the foundation of pruning-based NAS algorithms like NetAdapt. It decouples the search logic, which occurs in the cloud, from the model training, which occurs on the decentralized clients.
The framework operates through a series of iterations and rounds. In the cloud, a pre-trained model undergoes block-based DNN compression to generate multiple candidate architectures. These candidates are then sent to clients for training and testing. The cloud aggregates the results to pick the highest accuracy candidates, continuing the cycle until the resource budget is met.
Optimizing for Mobile Resources
To achieve efficiency, FedNAS incorporates three key optimizations:
- Parallel Candidate Training: Instead of requiring every client to train every model candidate, FedNAS partitions clients into groups. Each group focuses on a different architecture candidate, allowing for simultaneous evaluation of multiple designs.
- Dynamic Round Numbers: Recognizing that a perfectly trained model isn't necessary to identify poor designs, FedNAS uses fewer training rounds in early iterations, increasing them only as the search narrows.
- Early Dropping: If a candidate significantly underperforms after initial rounds, it is discarded immediately, preventing the waste of mobile resources on "dead-end" architectures.
To handle non-IID (non-independent and identically distributed) data, a principled client partition algorithm ensures each group is a representative "mini-mirror" of the total dataset using Manhattan distance metrics.
Evaluation and Results
The researchers evaluated FedNAS using ImageNet (IID) and Celeba (Non-IID) benchmarks. The results demonstrate that FedNAS achieves accuracy comparable to centralized state-of-the-art NAS while offering massive efficiency gains:
- Accuracy: On the Celeba dataset, FedNAS produced models that were 2.5x less complex than standard baselines while maintaining the same accuracy.
- Speed and Cost: Compared to a naive federated NAS implementation, FedNAS reduced communication costs by 277x and computation time by 281x on ImageNet.
- Robustness: The client partitioning strategy proved effective even with highly skewed data distributions.
Conclusion
FedNAS represents a significant step toward bringing automated intelligence to the edge without compromising user privacy. By leveraging the "Good Enough" principle—accepting lower-fidelity accuracy estimates early in the search—FedNAS can discard 90% of the search space at a fraction of the traditional cost. Future work will look toward searching architectures from scratch and further refining privacy protections regarding data distribution metadata.
Summary updated