BOASF: A Unified Framework for Speeding up Automatic Machine Learning via Adaptive Successive Filtering

Guanghui Zhu, Xin Fang, Lei Wang, Wenzhong Chen, Rong Gu, Chunfeng Yuan, Yihua HuangarXiv 2025 · DOI · OpenReview

TL;DR

This paper introduces BOASF, a novel framework that unifies model selection and hyperparameter optimization for automatic machine learning (AutoML). By combining Bayesian Optimization with an Adaptive Successive Filtering strategy, BOASF efficiently prunes underperforming models and intelligently allocates computational resources, leading to faster and more robust AutoML performance.

What Problem Does This Paper Address?

The increasing complexity and vastness of machine learning models and their hyperparameter spaces make it incredibly challenging for practitioners, especially those without deep expertise, to effectively select the best model and tune its parameters. This process is often time-consuming and requires significant domain knowledge. Existing AutoML methods often tackle model selection and hyperparameter optimization separately or struggle with the high dimensionality of the combined search space. The paper aims to provide a unified, efficient, and automated solution that can handle both tasks effectively, even under limited computational budgets.

Methods

BOASF is built upon a multi-armed bandit framework, where each "arm" represents a candidate model or a subspace of hyperparameters. The core innovation lies in the combination of two key components:

  • Bayesian Optimization (BO): For each arm (model or hyperparameter subspace), BO is used to intelligently select promising configurations to evaluate. This is more efficient than random search, as it learns from past evaluations to guide future choices. The paper specifically mentions using the Tree Parzen Estimator (TPE) for this purpose.
  • Adaptive Successive Filtering (ASF): This component dynamically prunes underperforming arms. After each evaluation round, BOASF calculates a Gaussian Upper Confidence Bound (UCB) for each arm based on its historical performance. This UCB score is then used to determine the probability of an arm advancing to the next round. A Softmax function is employed to adaptively allocate computational resources to the promising arms, with arms having a higher probability of advancing receiving more resources. This adaptive filtering and resource allocation is a key differentiator from methods like Hyperband, which use fixed ratios for discarding arms.

Datasets

The paper evaluated BOASF on a variety of datasets:

  • Model Selection: 24 datasets from the OpenML repository (binary and multi-class classification, <50,000 samples, no missing values) and 6 commonly-used datasets from UCI, LibSVM, and KDDCup.
  • Hyperparameter Optimization: The paper mentions using datasets for logistic regression and random forest models, but specific dataset names are not listed in this section.

Key Insights and Ideas

  • Unified Framework: The most significant insight is the successful unification of model selection and hyperparameter optimization within a single, coherent framework. This simplifies the AutoML process.
  • Adaptive Filtering is Crucial: The adaptive nature of ASF, which dynamically discards poor performers and allocates resources based on performance, is key to BOASF's efficiency. This avoids the potential pitfalls of fixed-ratio discarding methods.
  • BO for Configuration Selection: Leveraging Bayesian Optimization within each arm's evaluation significantly improves the chances of finding good configurations compared to random sampling.
  • Sub-space Partitioning for Hyperparameter Optimization: For hyperparameter optimization, BOASF tackles the high dimensionality by partitioning the search space into subspaces, treating each subspace as an arm. This makes the problem tractable.
  • Robust Anytime Performance: BOASF demonstrates strong performance across different time budgets, meaning it can deliver good results even when computational resources are limited.

Key Results

  • Model Selection: BOASF statistically outperformed AutoSklearn on 20/30 datasets with a 2-hour time budget and on 23/30 datasets with a 4-hour time budget. It also showed better performance than SelectBest and RandomForest&TPE.
  • Hyperparameter Optimization: BOASF achieved better balanced accuracy scores than Hyperopt-full and BOHB on both logistic regression and random forest models across different time limits.
  • Speedup: The adaptive filtering and resource allocation strategy significantly speeds up the AutoML process compared to existing methods.
  • Robustness: BOASF exhibited stable performance improvements and robust anytime performance under various time budgets.

Limitations and Future Work

  • UCB Score Range: The paper notes that the UCB score might need readjustment if the reward range is not between 0 and 1, as the current implementation relies on this.
  • Rescaling Method: While MinMaxScaler is used for scaling UCB scores to probabilities, the authors suggest that other rescaling methods with similar properties could also be employed.
  • TPE Comparison: The paper used TPE for BO within each arm and suggests comparing it with other Bayesian Optimization methods for potential further improvements.
  • Extension to Other Domains: The framework's adaptability suggests potential for extension to other machine learning tasks beyond classification.

Summary updated

Abstract

Machine learning has been making great success in many application areas. However, for the non-expert practitioners, it is always very challenging to address a machine learning task successfully and efficiently. Finding the optimal machine learning model or the hyperparameter combination set from a large number of possible alternatives usually requires considerable expert knowledge and experience. To tackle this problem, we propose a combined Bayesian Optimization and Adaptive Successive Filtering algorithm (BOASF) under a unified multi-armed bandit framework to automate the model selection or the hyperparameter optimization. Specifically, BOASF consists of multiple evaluation rounds in each of which we select promising configurations for each arm using the Bayesian optimization. Then, ASF can early discard the poor-performed arms adaptively using a Gaussian UCB-based probabilistic model. Furthermore, a Softmax model is employed to adaptively allocate available resources for each promising arm that advances to the next round. The arm with a higher probability of advancing will be allocated more resources. Experimental results show that BOASF is effective for speeding up the model selection and hyperparameter optimization processes while achieving robust and better prediction performance than the existing state-of-the-art automatic machine learning methods. Moreover, BOASF achieves better anytime performance under various time budgets.
BibTeX
@misc{zhu2025boasf,
  author = {Guanghui Zhu and Xin Fang and Lei Wang and Wenzhong Chen and Rong Gu and Chunfeng Yuan and Yihua Huang},
  title = {{BOASF: A Unified Framework for Speeding up Automatic Machine Learning via Adaptive Successive Filtering}},
  year = {2025},
  eprint = {2507.20446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.20446},
}
Figure