VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition

Yang Li, Yu Shen, Wentao Zhang, Jiawei Jiang, Bolin Ding, Yaliang Li, Jingren Zhou, Zhi Yang, Wentao Wu, Ce Zhang, Bin Cui

Abstract

End-to-end AutoML has attracted intensive interests from both academia and industry, which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning. Existing AutoML systems, however, suffer from scalability issues when applying to application domains with large, high-dimensional search spaces. We present VolcanoML, a scalable and extensible framework that facilitates systematic exploration of large AutoML search spaces. VolcanoML introduces and implements basic building blocks that decompose a large search space into smaller ones, and allows users to utilize these building blocks to compose an execution plan for the AutoML problem at hand. VolcanoML further supports a Volcano-style execution model - akin to the one supported by modern database systems - to execute the plan constructed. Our evaluation demonstrates that, not only does VolcanoML raise the level of expressiveness for search space decomposition in AutoML, it also leads to actual findings of decomposition strategies that are significantly more efficient than the ones employed by state-of-the-art AutoML systems such as auto-sklearn.

Introduction

Automated Machine Learning (AutoML) aims to find the best possible ML pipeline—covering feature engineering, model selection, and hyperparameter tuning—without human intervention. However, as these pipelines become more complex, the "search space" (the total number of possible combinations) explodes. Existing AutoML systems often treat this as a single, massive optimization problem. This "joint" approach hits a scalability wall when users try to enrich the search space with domain-specific features or new models. VolcanoML addresses these challenges by borrowing a classic concept from database systems: the execution plan.

Methods: Structured Decomposition

The core innovation of VolcanoML is its structured decomposition of the search space. Instead of one giant optimizer, it uses a tree of Building Blocks, inspired by the "Volcano" query execution model used in relational databases.

The system utilizes three primary types of blocks:

  1. Joint Block: Uses standard Bayesian Optimization (specifically SMAC) to optimize a specific subspace.
  2. Conditioning Block: Splits the search space based on a categorical variable (like choosing between a Random Forest or an SVM). It treats this as a Multi-Armed Bandit (MAB) problem, quickly pruning away "arms" (algorithms) that perform poorly.
  3. Alternating Block: Partitions the space into two independent sets (e.g., feature engineering vs. hyperparameters) and optimizes them alternately, focusing resources where they yield the most improvement.
Figure 1: VolcanoML execution plan illustrating the composition of conditioning, alternating, and joint blocks to decompose the search space.

These blocks are composed into an execution plan. To run the search, the system recursively invokes a do_next! command from the root of the tree down to the leaves, allowing for a modular and highly extensible search process. This modularity allows for easy extension, such as adding "Embedding Selection" stages by simply adding a new node to the plan.

Evaluation and Key Results

The authors evaluated VolcanoML using a robust suite of benchmarks, including 60 real-world OpenML datasets, large-scale datasets (up to 110k samples), and Kaggle competition datasets. VolcanoML consistently outperformed industry standards. On 30 classification tasks, VolcanoML outperformed auto-sklearn and TPOT on 25 and 23 tasks, respectively.

Figure 2: Accuracy improvement of VolcanoML compared to baseline systems across multiple OpenML datasets.

In a "Large" search space with over 100 hyperparameters, VolcanoML achieved an average rank of 1.65, while auto-sklearn trailed at 3.57. Efficiency gains were particularly notable on large datasets; on the Higgs dataset, VolcanoML reached a 27.2% test error within 4 hours—a result that took other systems 24 hours to achieve. Furthermore, in head-to-head tests on Kaggle datasets, VolcanoML was comparable to or better than four leading commercial AutoML platforms from major cloud providers.

Conclusion and Future Work

VolcanoML demonstrates that searching for an ML pipeline is fundamentally similar to executing a complex database query. By applying database-style execution plans and pruning strategies, it brings order to high-dimensional optimization. While current plans are often manually designed, future work aims to develop a "Query Optimizer" for AutoML to automatically determine the best decomposition strategies based on dataset characteristics. The authors also see potential in applying "push-down" logic to handle constrained optimization problems more effectively.

Summary updated

Abstract

End-to-end AutoML has attracted intensive interests from both academia and industry, which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning. Existing AutoML systems, however, suffer from scalability issues when applying to application domains with large, high-dimensional search spaces. We present VolcanoML, a scalable and extensible framework that facilitates systematic exploration of large AutoML search spaces. VolcanoML introduces and implements basic building blocks that decompose a large search space into smaller ones, and allows users to utilize these building blocks to compose an execution plan for the AutoML problem at hand. VolcanoML further supports a Volcano-style execution model - akin to the one supported by modern database systems - to execute the plan constructed. Our evaluation demonstrates that, not only does VolcanoML raise the level of expressiveness for search space decomposition in AutoML, it also leads to actual findings of decomposition strategies that are significantly more efficient than the ones employed by state-of-the-art AutoML systems such as auto-sklearn.
BibTeX
@article{li2023volcanoml,
  author = {Yang Li and Yu Shen and Wentao Zhang and Jiawei Jiang and Bolin Ding and Yaliang Li and Jingren Zhou and Zhi Yang and Wentao Wu and Ce Zhang and Bin Cui},
  title = {{VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition}},
  journal = {VLDB J.},
  year = {2023},
  doi = {10.1007/s00778-022-00752-2},
}
Figure