FLAML: A Fast and Lightweight AutoML Library

Chi Wang, Qingyun Wu, Markus Weimer, Erkang ZhuMLSys 2021

TL;DR

This paper introduces FLAML, a new AutoML library designed for speed and efficiency, optimizing for low computational cost while finding accurate machine learning models. FLAML significantly outperforms existing AutoML solutions on a large benchmark, even with substantially smaller resource budgets.

What Problem Does This Paper Address?

Building ML-infused applications is becoming increasingly common, even for smaller teams without extensive ML expertise or massive computational resources. This necessitates Automated Machine Learning (AutoML) solutions that can efficiently find good models (learners and hyperparameters) for a given dataset and error metric, without requiring significant computational cost or time. Existing AutoML libraries often struggle with this, requiring long training times and large resource budgets, sometimes even underperforming simpler baselines. This paper addresses the need for an economical AutoML system that holistically considers the trade-off between trial cost and model error, making it suitable for resource-constrained environments.

Methods

FLAML is a lightweight Python library that employs a novel search strategy to optimize for both cost and error. Its core innovation lies in its fit() method, which adaptively selects learners, hyperparameters, sample sizes, and resampling strategies. Key aspects of its approach include:

  • Decoupled Search Variables and Strategies: FLAML separates the search for learners, hyperparameters, sample size, and resampling strategy, but couples the selection of hyperparameters and sample size to reflect their interdependencies.
  • Estimated Cost for Improvement (ECI): A novel metric used to estimate the cost required for a learner to achieve a better error than the current best. This metric guides the search by prioritizing configurations with lower estimated costs.
  • Probabilistic Learner Selection: Learners are chosen with a probability inversely proportional to their ECI, ensuring a balance between exploring promising learners and giving all learners a chance to be searched.
  • Randomized Direct Search for Hyperparameters: This technique efficiently explores the hyperparameter space, adapting step-size and using random restarts to escape local optima.
  • Adaptive Sample Size: FLAML dynamically adjusts the training data sample size, starting small and increasing it as needed, which helps in balancing cost and error estimation.
  • No Meta-Learning or Ensembling (by default): To maintain simplicity and usability for ad-hoc datasets, FLAML avoids meta-learning and ensembles by default, focusing on core search efficiency.

Datasets

The paper evaluates FLAML on a combination of datasets from:

  • AutoML Benchmark (classification): A collection of 39 classification tasks from previous AutoML papers, competitions, and benchmarks.
  • PMLB (regression): A benchmark suite for machine learning evaluation, from which 14 regression datasets with more than 10,000 instances were selected.

The authors note that the search spaces of the compared libraries are not uniform, making direct comparisons challenging but highlighting FLAML's adaptability.

Key Insights and Ideas

  • Cost-Error Tradeoff is Crucial: Simply optimizing for low error can lead to prohibitively expensive trials, while optimizing only for cost can result in poor models. FLAML's ECI metric effectively balances these.
  • Sample Size Matters: The choice of sample size significantly impacts both error and cost. FLAML's adaptive sample size strategy allows for faster initial exploration with smaller datasets and more thorough evaluation with larger ones.
  • Learner Prioritization is Key: By estimating the cost to improve, FLAML intelligently prioritizes which learners to explore, leading to faster convergence to good models.
  • Simplicity Can Be Powerful: FLAML's focus on efficient search strategies without relying on complex meta-learning or ensembles (by default) makes it highly practical for ad-hoc scenarios.
  • Adaptive Behavior is Essential: FLAML's ability to dynamically adjust its search based on observed costs and errors is a key factor in its strong performance, especially in early stages of the search.

Key Results

  • FLAML significantly outperforms top-ranked open-source AutoML libraries (Auto-sklearn, TPOT, H2O AutoML) and a commercial cloud-based AutoML service.
  • On a large open-source AutoML benchmark, FLAML achieved better or matching results in a majority of tasks, often with orders of magnitude smaller budget constraints.
  • For example, FLAML's 1-minute budget performance was comparable to or better than other libraries' 10-minute or even 1-hour budgets on a significant percentage of datasets.
  • In a selectivity estimation task for database systems, FLAML outperformed other AutoML libraries and a manual configuration.

Limitations and Future Work

  • No Default Meta-Learning or Ensembling: While FLAML is designed for ad-hoc datasets, the authors suggest that incorporating lightweight meta-learning and ensemble techniques could be valuable future work, provided they maintain the system's economic nature.
  • Search Space Decomposition: FLAML decomposes the search space, which is efficient but might miss complex interactions between certain hyperparameters if they are not in the same decomposed subspace.
  • Future Work: The authors suggest exploring meta-learning for per-task instance optimization and investigating the tradeoff between cost and error when model ensembles are introduced. They also mention the possibility of extending FLAML to work with multiple search threads in parallel when abundant cores are available.

Summary updated

BibTeX
@inproceedings{wang2021flaml,
  author = {Chi Wang and Qingyun Wu and Markus Weimer and Erkang Zhu},
  title = {{FLAML: A Fast and Lightweight AutoML Library}},
  booktitle = {MLSys},
  year = {2021},
}
Figure