Introduction
In modern machine learning, designing optimal neural network architectures (Neural Architecture Search, or NAS) and tuning training configurations (Hyperparameter Optimization, or HO) are critical yet labor-intensive tasks. These processes are deeply interdependent; for instance, the ideal learning rate often depends on the specific network depth or complexity.
This challenge is significantly compounded in Federated Learning (FL) environments. In FL, data is distributed across numerous edge devices and cannot be centralized due to privacy regulations like GDPR. Existing FL frameworks typically address either NAS or HO in isolation, often neglecting the communication overhead of sequential optimization or failing to provide rigorous privacy guarantees.
The FEATHERS Framework
To bridge this gap, we introduce FEATHERS (FEderated ArchiTecture and HypERparameter Search). FEATHERS is the first framework to jointly optimize architectures and hyperparameters in a federated setting while maintaining formal Differential Privacy (DP) guarantees.
The framework operates through a two-phase alternating optimization strategy:
- Hyperparameter Optimization (HO) Phase: The server treats hyperparameter selection as a multi-armed bandit problem. It samples configurations ("arms"), and clients return a "reward" based on validation loss improvement. An $\epsilon$-greedy strategy balances exploration of new settings with exploitation of known high-performers.
- Neural Architecture Search (NAS) Phase: Utilizing a differentiable approach inspired by DARTS, the architecture is represented by learnable parameters within "cells." These are updated via local gradient descent on clients and aggregated using Federated Averaging (FedAvg).
- Privacy Preservation: FEATHERS integrates DP-SGD, clipping gradients and adding Gaussian noise to model weights, architecture parameters, and even the rewards in the HO phase to prevent sensitive data leakage through side channels.
Experimental Evaluation
FEATHERS was evaluated across diverse benchmarks, including image classification tasks on Fashion-MNIST, CIFAR-10, and Tiny-Imagenet, as well as a massive, highly imbalanced tabular dataset for fraud detection. The experiments tested the framework's robustness against data skew (non-i.i.d. settings) and its scalability as the number of clients increased from 5 to 100.
Key Results and Insights
FEATHERS consistently outperformed established baselines such as DARTS and FedEx. Notably, it maintained high performance even under strict privacy constraints ($\epsilon \ge 1$) and showed superior scalability in large-scale distributed systems.
Key observations include:
- Adaptive Learning: The bandit mechanism naturally favors lower learning rates as training progresses, mimicking an automated annealing process that stabilizes convergence.
- Efficiency: By alternating NAS and HO, FEATHERS avoids the prohibitive cost of full NAS cycles for every hyperparameter set, adding only minimal overhead (approx. 0.4 GPU-days) compared to standard DARTS.
- Robustness: The joint optimization approach finds architectures that generalize effectively across diverse, non-i.i.d. local datasets.
Conclusion and Future Work
FEATHERS demonstrates that joint architecture and hyperparameter search is not only feasible in federated learning but also highly effective when combined with differential privacy. Future research will explore continuous hyperparameter search spaces and stateful bandit mechanisms to further refine the optimization process in massive, cross-device federated networks.
Summary updated