Single-shot General Hyper-parameter Optimization for Federated Learning

Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, Heiko LudwigICLR 2023 · OpenReview

TL;DR

Hyper-parameter optimization (HPO) is the "secret sauce" of machine learning, but in Federated Learning (FL), it’s usually a communication nightmare. This paper introduces FLoRA (Federated Loss SuRface Aggregation), a framework that identifies high-performing hyper-parameters in a "single-shot" manner—saving up to 39x in communication costs while outperforming standard baselines across diverse models like Gradient Boosted Trees and SVMs.

What Problem Does This Paper Address?

In traditional centralized learning, we find the best hyper-parameters (HPs)—like learning rates or tree depths—by training dozens of models on a central server. In Federated Learning, data stays on local devices (parties), and training even a single model requires many rounds of expensive communication between those parties and a central server.

Existing FL-HPO methods face three major hurdles:

  1. Communication Overhead: Most methods are "multi-shot," meaning they require training multiple global models to find the best HPs, which is prohibitively slow and expensive.
  2. Model Limitations: Many current solutions only work for neural networks (using "weight sharing"), leaving out traditional models like Gradient Boosted Decision Trees (GBDTs) that are essential for tabular data in industries like finance and healthcare.
  3. Data Heterogeneity: FL data is often "Non-IID" (not independently and identically distributed). A hyper-parameter that works well for Party A might be a disaster for Party B.

Methods

The authors propose FLoRA, a "single-shot" framework. Instead of training multiple global models, FLoRA follows a clever three-step process:

  1. Asynchronous Local HPO: Each party independently runs HPO on its own local data (using techniques like Bayesian Optimization) to find which HPs work best for them.
  2. Loss Surface Aggregation: Each party sends their (Hyper-parameter, Loss) pairs to the central aggregator. The aggregator then builds a "global loss surface" by training a regressor (like a Random Forest) to predict the loss for any given HP set.
  3. Single-Shot Selection: The aggregator identifies the HP set that minimizes the predicted global loss and initiates exactly one full federated training run using those HPs.
Aggregation Strategies

The paper explores four ways to aggregate these local "surfaces":

  • SGM (Single Global Model): Merging all data into one regressor.
  • SGM+U: Adding an uncertainty penalty to avoid unexplored regions.
  • MPLM (Maximum of Per-party Local Models): A pessimistic approach that only favors HPs that perform well for everyone.
  • APLM (Average of Per-party Local Models): Averaging the predicted losses from each party's local regressor.

Datasets

The researchers evaluated FLoRA using seven binary classification datasets from OpenML: EEG eye state, Electricity, Heart statlog, Oil spill, Pollen, Sonar, and PC3. These datasets vary significantly in size (from 208 to 45,312 rows) and feature counts, providing a robust test for tabular data performance. They also used MNIST to specifically simulate extreme data heterogeneity (Non-IID settings).

Key Insights and Ideas

  • The "Consensus" HP: The core "aha moment" is that you don't need to train a global model to know if an HP is good. If an HP set performs well across many different local data distributions independently, it is a statistically strong candidate for the global federated model.
  • Pessimism is a Virtue: In highly heterogeneous (Non-IID) settings, the MPLM (Maximum) and APLM (Average) strategies are superior. By looking for HPs that satisfy the "worst-off" party or the average, the system avoids HPs that overfit to a single party's unique data.
  • Theoretical Grounding: The authors provide the first rigorous theoretical analysis of the "optimality gap" in FL-HPO, proving that the error is bounded by a combination of local HPO quality and the "Wasserstein distance" (a measure of how different the parties' data distributions are).

Key Results

  • Massive Efficiency: To reach the same accuracy as FLoRA, a standard "multi-shot" approach would require 39x more communication rounds for Multi-layer Perceptrons and 24x more for Gradient Boosted Trees.
Efficiency Comparison
  • Superior Accuracy: FLoRA consistently outperformed the "single-shot baseline" (using default scikit-learn HPs). In many cases, it achieved near-optimal performance (zero regret) compared to a theoretical centralized model.
  • Robustness: Even as the number of parties increased (up to 100) and data became more heterogeneous, FLoRA (specifically the APLM variant) maintained high accuracy.

Limitations and Future Work

  • Aggregator HPs: FLoRA currently focuses on model HPs (like tree depth). It does not yet handle HPs specific to the FL process itself, such as the weights used during the aggregation step (e.g., FedAvg weights).
  • Inactive HPs: The method struggles with HPs that are "inactive" during local training but become relevant during global aggregation.
  • Future Directions: The authors suggest extending FLoRA to the "CASH" problem (Combined Algorithm Selection and Hyper-parameter optimization), where the system chooses both the best model type and its HPs simultaneously. They also aim to explore "few-shot" versions for even more complex HP spaces.

Summary updated

Abstract

We address the problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO). We introduce Federated Loss SuRface Aggregation (FLoRA), a general FL-HPO solution framework that can address use cases of tabular data and any Machine Learning (ML) model including gradient boosting training algorithms, SVMs, neural networks, among others and thereby further expands the scope of FL-HPO. FLoRA enables single-shot FL-HPO: identifying a single set of good hyper-parameters that are subsequently used in a single FL training. Thus, it enables FL-HPO solutions with minimal additional communication overhead compared to FL training without HPO. Utilizing standard smoothness assumptions, we theoretically characterize the optimality gap of FLoRA for any convex and non-convex loss functions, which explicitly accounts for the heterogeneous nature of the parties' local data distributions, a dominant characteristic of FL systems. Our empirical evaluation of FLoRA for multiple FL algorithms on seven OpenML datasets demonstrates significant model accuracy improvements over the baselines, and robustness to increasing number of parties involved in FL-HPO training.
BibTeX
@inproceedings{zhou2023single,
  author = {Yi Zhou and Parikshit Ram and Theodoros Salonidis and Nathalie Baracaldo and Horst Samulowitz and Heiko Ludwig},
  title = {{Single-shot General Hyper-parameter Optimization for Federated Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=3RhuF8foyPW},
}
Figure