HANF: Hyperparameter And Neural Architecture Search in Federated Learning

Jonas Seng, Pooja Prasad, Devendra Singh Dhami, Kristian KerstingarXiv 2022 · DOI · arXiv

Introduction

In modern machine learning, designing optimal neural network architectures (Neural Architecture Search, or NAS) and tuning training configurations (Hyperparameter Optimization, or HO) are critical yet labor-intensive tasks. These processes are deeply interdependent; for instance, the ideal learning rate often depends on the specific network depth or complexity.

This challenge is significantly compounded in Federated Learning (FL) environments. In FL, data is distributed across numerous edge devices and cannot be centralized due to privacy regulations like GDPR. Existing FL frameworks typically address either NAS or HO in isolation, often neglecting the communication overhead of sequential optimization or failing to provide rigorous privacy guarantees.

Comparison of FEATHERS with existing methods across NAS, HO, Differential Privacy (DP), and Federated Learning (Fed) capabilities.

The FEATHERS Framework

To bridge this gap, we introduce FEATHERS (FEderated ArchiTecture and HypERparameter Search). FEATHERS is the first framework to jointly optimize architectures and hyperparameters in a federated setting while maintaining formal Differential Privacy (DP) guarantees.

The framework operates through a two-phase alternating optimization strategy:

  1. Hyperparameter Optimization (HO) Phase: The server treats hyperparameter selection as a multi-armed bandit problem. It samples configurations ("arms"), and clients return a "reward" based on validation loss improvement. An $\epsilon$-greedy strategy balances exploration of new settings with exploitation of known high-performers.
  2. Neural Architecture Search (NAS) Phase: Utilizing a differentiable approach inspired by DARTS, the architecture is represented by learnable parameters within "cells." These are updated via local gradient descent on clients and aggregated using Federated Averaging (FedAvg).
  3. Privacy Preservation: FEATHERS integrates DP-SGD, clipping gradients and adding Gaussian noise to model weights, architecture parameters, and even the rewards in the HO phase to prevent sensitive data leakage through side channels.

Experimental Evaluation

FEATHERS was evaluated across diverse benchmarks, including image classification tasks on Fashion-MNIST, CIFAR-10, and Tiny-Imagenet, as well as a massive, highly imbalanced tabular dataset for fraud detection. The experiments tested the framework's robustness against data skew (non-i.i.d. settings) and its scalability as the number of clients increased from 5 to 100.

Key Results and Insights

FEATHERS consistently outperformed established baselines such as DARTS and FedEx. Notably, it maintained high performance even under strict privacy constraints ($\epsilon \ge 1$) and showed superior scalability in large-scale distributed systems.

Performance comparison (Accuracy/F1-score) of FEATHERS against DARTS and FedEx across multiple datasets and client counts.

Key observations include:

  • Adaptive Learning: The bandit mechanism naturally favors lower learning rates as training progresses, mimicking an automated annealing process that stabilizes convergence.
  • Efficiency: By alternating NAS and HO, FEATHERS avoids the prohibitive cost of full NAS cycles for every hyperparameter set, adding only minimal overhead (approx. 0.4 GPU-days) compared to standard DARTS.
  • Robustness: The joint optimization approach finds architectures that generalize effectively across diverse, non-i.i.d. local datasets.

Conclusion and Future Work

FEATHERS demonstrates that joint architecture and hyperparameter search is not only feasible in federated learning but also highly effective when combined with differential privacy. Future research will explore continuous hyperparameter search spaces and stateful bandit mechanisms to further refine the optimization process in massive, cross-device federated networks.

Summary updated

Abstract

Deep neural architectures have profound impact on achieved performance in many of today's AI tasks, yet, their design still heavily relies on human prior knowledge and experience. Neural architecture search (NAS) together with hyperparameter optimization (HO) helps to reduce this dependence. However, state of the art NAS and HO rapidly become infeasible with increasing amount of data being stored in a distributed fashion, typically violating data privacy regulations such as GDPR and CCPA. As a remedy, we introduce FEATHERS - $\textbf{FE}$derated $\textbf{A}$rchi$\textbf{T}$ecture and $\textbf{H}$yp$\textbf{ER}$parameter $\textbf{S}$earch, a method that not only optimizes both neural architectures and optimization-related hyperparameters jointly in distributed data settings, but further adheres to data privacy through the use of differential privacy (DP). We show that FEATHERS efficiently optimizes architectural and optimization-related hyperparameters alike, while demonstrating convergence on classification tasks at no detriment to model performance when complying with privacy constraints.
BibTeX
@misc{seng2022hanf,
  author = {Jonas Seng and Pooja Prasad and Devendra Singh Dhami and Kristian Kersting},
  title = {{HANF: Hyperparameter And Neural Architecture Search in Federated Learning}},
  year = {2022},
  eprint = {2206.12342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.12342},
}
Figure