Efficient End-to-End AutoML via Scalable Search Space Decomposition

Yang Li, Yu Shen, Wentao Zhang, Ce Zhang, Bin Cui

TL;DR

VolcanoML is a scalable AutoML framework that tackles the "curse of dimensionality" in machine learning pipeline search by decomposing massive search spaces into manageable sub-tasks. Inspired by database query execution models, it uses a modular "building block" approach to outperform state-of-the-art systems like auto-sklearn, achieving up to 11x speedups on large-scale datasets.

What Problem Does This Paper Address?

Automated Machine Learning (AutoML) aims to find the best end-to-end pipeline—covering feature engineering (FE), algorithm selection, and hyperparameter optimization (HPO)—without human intervention. However, as we add more domain-specific features and complex models, the "joint search space" (the total combination of all possible choices) explodes.

Existing systems often treat this as one giant black-box optimization problem. This leads to two major issues:

  1. Scalability: Jointly optimizing dozens of hyperparameters alongside complex feature transformations is computationally exhausting.
  2. Extensibility: It is often difficult for users to plug in domain-specific operators (like custom normalization for astronomy data) without breaking the underlying search algorithm.

Methods

The authors introduce VolcanoML, which treats search space decomposition as an execution plan, much like how a database engine breaks down a SQL query.

The Building Blocks

Instead of one big search, VolcanoML uses three types of nodes to build an execution tree:

  • Joint Block: The standard approach. It uses Bayesian Optimization to search a specific subspace.
  • Conditioning Block: Partitions the space based on a categorical choice (e.g., "Which algorithm should I use?"). It uses Multi-Armed Bandit (MAB) logic to play different "arms" (subspaces) and prune those that perform poorly.
  • Alternating Block: Decomposes the space into two parts (e.g., Features vs. Hyperparameters) and optimizes them alternately, similar to coordinate descent.
Different ways to decompose the joint search space into strategies.

The Volcano Execution Model

Borrowing from the classic "Volcano" iterator model in databases, each block implements a do_next() interface. The root of the tree calls do_next(), which recursively triggers child blocks to perform one iteration of optimization. This allows for complex, nested search strategies that are easy to manage and extend.

Meta-Learning Acceleration

To avoid starting from scratch on every new dataset, VolcanoML uses:

  • RankNet: To predict which algorithms are likely to work best based on dataset "meta-features."
  • RGPE (Ranking-weighted Gaussian Process Ensemble): To leverage historical optimization data from previous tasks to guide the current search.

Datasets

The framework was evaluated against a massive benchmark suite:

  • OpenML: 60 real-world datasets (40 classification, 20 regression).
  • Large-Scale Data: 10 massive datasets (e.g., Higgs, Mnist_784) with up to 1.1 million samples.
  • Kaggle: 6 diverse competition datasets to compare against industrial platforms.
  • Meta-learning: Trained on results from 140 OpenML tasks.

Key Insights and Ideas

  • Decomposition > Joint Search: The core "aha!" moment is that the improvements from feature engineering and HPO are often complementary. By alternating between them rather than searching them jointly, the system finds high-quality pipelines much faster.
  • Structural Flexibility: Because the search is a tree of blocks, users can easily enrich the search space. For example, the authors successfully added an "embedding selection" stage for image data using pre-trained models from TensorFlow Hub.
  • Pruning is Essential: Using Multi-Armed Bandits in the Conditioning Block allows the system to quickly stop wasting time on algorithms that are clearly unsuited for a specific dataset.
The standard feature engineering pipeline supported by VolcanoML.

Key Results

  • Superior Performance: VolcanoML outperformed auto-sklearn and TPOT on 25 out of 30 classification tasks and 17 out of 20 regression tasks.
  • Massive Speedups: On the Higgs dataset, VolcanoML reached a lower test error in just 4 hours than its competitors could achieve in 24 hours. Overall, it showed a 4.3x to 11x speedup in reaching target validation errors.
  • Industrial Strength: VolcanoML proved competitive with—and often outperformed—commercial AutoML services from Google Cloud, AWS, Azure, and Oracle across six Kaggle competitions.
  • Enrichment Benefits: Adding a single "SMOTE" balancer operator to the search space improved accuracy on imbalanced datasets by up to 3.57%.

Limitations and Future Work

  • Plan Generation Overhead: While the paper explores automatic execution plan generation, searching for the best tree structure can itself be time-consuming.
  • Coarse Granularity: The current decomposition is mostly at the task level (e.g., FE vs. HPO). Finer-grained partitioning (e.g., different plans for different subsets of features) remains an open challenge.
  • Reinforcement Learning: The authors suggest that future work could use RL to dynamically adapt the execution plan during the search process, rather than relying on a fixed tree structure.

Summary updated

Abstract

End-to-end AutoML has attracted intensive interests from both academia and industry which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning. Existing AutoML systems, however, suffer from scalability issues when applying to application domains with large, high-dimensional search spaces. We present VolcanoML, a scalable and extensible framework that facilitates systematic exploration of large AutoML search spaces. VolcanoML introduces and implements basic building blocks that decompose a large search space into smaller ones, and allows users to utilize these building blocks to compose an execution plan for the AutoML problem at hand. VolcanoML further supports a Volcano-style execution model -- akin to the one supported by modern database systems -- to execute the plan constructed. Our evaluation demonstrates that, not only does VolcanoML raise the level of expressiveness for search space decomposition in AutoML, it also leads to actual findings of decomposition strategies that are significantly more efficient than the ones employed by state-of-the-art AutoML systems such as auto-sklearn. This paper is the extended version of the initial VolcanoML paper appeared in VLDB 2021.
BibTeX
@article{li2023efficient,
  author = {Yang Li and Yu Shen and Wentao Zhang and Ce Zhang and Bin Cui},
  title = {{Efficient End-to-End AutoML via Scalable Search Space Decomposition}},
  journal = {VLDB J.},
  year = {2023},
  doi = {10.1007/s00778-022-00752-2},
}
Figure