TL;DR
This paper introduces BOASF, a novel framework that unifies model selection and hyperparameter optimization for automatic machine learning (AutoML). By combining Bayesian Optimization with an Adaptive Successive Filtering strategy, BOASF efficiently prunes underperforming models and intelligently allocates computational resources, leading to faster and more robust AutoML performance.
What Problem Does This Paper Address?
The increasing complexity and vastness of machine learning models and their hyperparameter spaces make it incredibly challenging for practitioners, especially those without deep expertise, to effectively select the best model and tune its parameters. This process is often time-consuming and requires significant domain knowledge. Existing AutoML methods often tackle model selection and hyperparameter optimization separately or struggle with the high dimensionality of the combined search space. The paper aims to provide a unified, efficient, and automated solution that can handle both tasks effectively, even under limited computational budgets.
Methods
BOASF is built upon a multi-armed bandit framework, where each "arm" represents a candidate model or a subspace of hyperparameters. The core innovation lies in the combination of two key components:
- Bayesian Optimization (BO): For each arm (model or hyperparameter subspace), BO is used to intelligently select promising configurations to evaluate. This is more efficient than random search, as it learns from past evaluations to guide future choices. The paper specifically mentions using the Tree Parzen Estimator (TPE) for this purpose.
- Adaptive Successive Filtering (ASF): This component dynamically prunes underperforming arms. After each evaluation round, BOASF calculates a Gaussian Upper Confidence Bound (UCB) for each arm based on its historical performance. This UCB score is then used to determine the probability of an arm advancing to the next round. A Softmax function is employed to adaptively allocate computational resources to the promising arms, with arms having a higher probability of advancing receiving more resources. This adaptive filtering and resource allocation is a key differentiator from methods like Hyperband, which use fixed ratios for discarding arms.
Datasets
The paper evaluated BOASF on a variety of datasets:
- Model Selection: 24 datasets from the OpenML repository (binary and multi-class classification, <50,000 samples, no missing values) and 6 commonly-used datasets from UCI, LibSVM, and KDDCup.
- Hyperparameter Optimization: The paper mentions using datasets for logistic regression and random forest models, but specific dataset names are not listed in this section.
Key Insights and Ideas
- Unified Framework: The most significant insight is the successful unification of model selection and hyperparameter optimization within a single, coherent framework. This simplifies the AutoML process.
- Adaptive Filtering is Crucial: The adaptive nature of ASF, which dynamically discards poor performers and allocates resources based on performance, is key to BOASF's efficiency. This avoids the potential pitfalls of fixed-ratio discarding methods.
- BO for Configuration Selection: Leveraging Bayesian Optimization within each arm's evaluation significantly improves the chances of finding good configurations compared to random sampling.
- Sub-space Partitioning for Hyperparameter Optimization: For hyperparameter optimization, BOASF tackles the high dimensionality by partitioning the search space into subspaces, treating each subspace as an arm. This makes the problem tractable.
- Robust Anytime Performance: BOASF demonstrates strong performance across different time budgets, meaning it can deliver good results even when computational resources are limited.
Key Results
- Model Selection: BOASF statistically outperformed AutoSklearn on 20/30 datasets with a 2-hour time budget and on 23/30 datasets with a 4-hour time budget. It also showed better performance than SelectBest and RandomForest&TPE.
- Hyperparameter Optimization: BOASF achieved better balanced accuracy scores than Hyperopt-full and BOHB on both logistic regression and random forest models across different time limits.
- Speedup: The adaptive filtering and resource allocation strategy significantly speeds up the AutoML process compared to existing methods.
- Robustness: BOASF exhibited stable performance improvements and robust anytime performance under various time budgets.
Limitations and Future Work
- UCB Score Range: The paper notes that the UCB score might need readjustment if the reward range is not between 0 and 1, as the current implementation relies on this.
- Rescaling Method: While MinMaxScaler is used for scaling UCB scores to probabilities, the authors suggest that other rescaling methods with similar properties could also be employed.
- TPE Comparison: The paper used TPE for BO within each arm and suggests comparing it with other Bayesian Optimization methods for potential further improvements.
- Extension to Other Domains: The framework's adaptability suggests potential for extension to other machine learning tasks beyond classification.
Summary updated