Introduction
In standard machine learning, Automated Machine Learning (AutoML) is used to solve the Combined Algorithm Selection and Hyperparameter optimization (CASH) problem—automatically selecting the optimal model (e.g., Random Forest vs. XGBoost) and its corresponding settings. However, applying CASH to Federated Learning (FL) presents a significant challenge. In FL, data remains decentralized on private devices, and performing a full training cycle for every possible algorithm-hyperparameter combination would lead to an unsustainable explosion in communication and computation costs.
FLASH is the first framework designed to tackle the CASH problem specifically for FL. By utilizing a multi-fidelity approach, FLASH prunes underperforming algorithms early and evaluates hyperparameters locally, achieving performance close to centralized AutoML while maintaining efficiency.
Methodology
FLASH treats the CASH problem as a bi-level optimization task consisting of algorithm selection and hyperparameter optimization (HPO).
1. Algorithm Selection: The "Fail Fast" Approach
FLASH employs a multi-fidelity incremental approach to algorithm selection. Instead of training on the full dataset, it begins with small fractions of client data and calculates a "Loss Projection" to estimate final performance. If an algorithm's projected performance falls below a leader by a specific tolerance threshold ($\Delta$), it is discarded early.
As shown in the table below, the choice of $\Delta$ allows researchers to balance accuracy and resource consumption. Increasing the threshold allows for more thorough exploration (reducing error) at the cost of higher training overhead.
2. Inner Level: FL-HPO
To optimize hyperparameters without exhaustive FL training rounds, FLASH utilizes three decentralized strategies:
- Local Best Model (LBM): Clients identify their best local settings, which the server then aggregates.
- Local K-Best Model (LKBM): Clients share their top $K$ settings for cross-validation across the network.
- Regression-based Model (RM): The server trains a surrogate model (e.g., Random Forest) using client-reported (hyperparameter, loss) pairs to predict performance across the search space.
Crucially, FLASH only performs a single full FL training round at the very end of the process once the optimal pair is identified.
Key Results and Insights
The researchers evaluated FLASH across eight diverse datasets (including EEG-Eye, Higgs, and Magic Telescope) against seven popular machine learning algorithms.
Near-Centralized Performance
FLASH variants consistently achieved validation accuracies within 1.5% of centralized CASH methods (like Auto-SKL) that have access to all data in a single location. Furthermore, FLASH significantly outperformed "CASH-D" (using default scikit-learn settings), proving that automated tuning is both viable and necessary in federated environments.
Efficiency and the "Non-IID Tax"
The study found that local evaluations on client subsets serve as remarkably good proxies for global performance, provided the data is not extremely heterogeneous. However, the paper notes a "Non-IID Tax": as client data becomes more dissimilar, the gap between FLASH and centralized CASH grows, as local snapshots become less representative of the global distribution.
Conclusion and Future Work
FLASH demonstrates that the CASH problem can be solved efficiently in federated settings by pruning the search space early and leveraging local client computation. While highly effective for standard machine learning algorithms, future work will look to extend this framework to Neural Architecture Search (NAS) and further optimize communication rounds for extremely heterogeneous (non-IID) data environments.
Summary updated