VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition

Yang Li, Yu Shen, Wentao Zhang, Jiawei Jiang, Yaliang Li, Bolin Ding, Jingren Zhou, Zhi Yang, Wentao Wu, Ce Zhang, Bin Cui

TL;DR

Existing end-to-end AutoML systems struggle to scale when user-defined or domain-specific operations expand the search space across feature engineering, model selection, and hyperparameter tuning. Inspired by classic database query engines, VolcanoML introduces a modular framework that decomposes large AutoML search spaces into tree-structured execution plans of smaller subproblems. By combining Bayesian optimization with bandit-driven conditioning and alternating execution, VolcanoML navigates high-dimensional spaces significantly faster and more effectively than state-of-the-art tools like auto-sklearn and TPOT.

What Problem Does This Paper Address?

Building an end-to-end machine learning pipeline requires searching a massive, composite space that covers feature preprocessing, algorithm selection, and hyperparameter tuning. Most popular AutoML frameworks (such as auto-sklearn and TPOT) treat this entire pipeline as a single, joint black-box optimization problem.

In practice, users frequently need to enrich the default search space with domain-specific components—such as specialized feature normalizers, pre-trained embeddings (e.g., from TensorFlow Hub), or custom loss functions. Joint optimization over this expanded space hits an immediate scalability bottleneck: standard Bayesian optimization (BO) degrades rapidly in high dimensions. While some prior systems decomposed isolated stages (like algorithm selection from tuning), no principled abstraction existed to decompose the entire end-to-end AutoML pipeline into manageable, composable pieces.

Methods

VolcanoML models search space decomposition analogously to physical query plans in relational database management systems:

  • Modular Building Blocks: Search subproblems are modeled as building blocks that expose standard interfaces (init, do_next!, get_current_best, and expected utility queries):
  • Joint Block: Solves a localized subspace using Bayesian Optimization (SMAC with Expected Improvement) or multi-fidelity early stopping (MFES-HB).
  • Conditioning Block: Partitions the search space by conditioning on a categorical variable (e.g., the ML algorithm). It treats each choice as an arm in a Multi-Armed Bandit (MAB) problem and prunes underperforming arms using confidence bounds on their extrapolated Expected Utility (EU).
  • Alternating Block: Splits variables into two subspaces (e.g., feature engineering vs. algorithm hyperparameters) and optimizes them alternately. It uses a "rotting bandit" strategy driven by Expected Utility Improvement (EUI) to dynamically allocate search budget to whichever subspace is yielding greater marginal gains.
  • Volcano-Style Execution: Execution plans form a tree of building blocks. Calling do_next! at the root recursively invokes child nodes down to the leaves, passing current best configurations back and forth across branches.
  • Extensibility & Meta-Learning: The framework easily incorporates pre-trained neural feature extractors and warm-starts building blocks using meta-learning across past task runs.
            [Condition on Algorithm]  <-- Bandit arm elimination
                   /        \
    [Alternating Block]   [Alternating Block]  <-- EUI-based switching
        /         \           /         \
  [FE Joint]  [HP Joint]  [FE Joint]  [HP Joint]  <-- SMAC / MFES-HB

Datasets

The authors evaluated VolcanoML across multiple benchmarks:

  • OpenML Repository (60 datasets):
  • 40 classification tasks (30 medium datasets with 1k–12k samples, 10 large datasets with 20k–110k samples including Mnist_784, Kropt, Electricity, and Higgs).
  • 20 regression tasks of medium size.
  • Kaggle Datasets (6 competitions): Influence Network, Virus Prediction, Employee Access, Customer Satisfaction, Business Value, and Flavours, used to benchmark against commercial cloud AutoML platforms.
  • Extensibility Benchmark: Kaggle dogs-vs-cats for multimodal/image embedding selection, along with 5 imbalanced OpenML datasets (sick, pc2, abalone, page-blocks, hypothyroid) to evaluate custom sampling operators.

Key Insights and Ideas

  • Decoupling Algorithms from Features: Feature preferences vary heavily by model family, but within a given model family, feature engineering and hyperparameter tuning provide largely complementary improvements. Factoring out the algorithm choice first, followed by alternating optimization between features and hyperparameters, avoids exploring unproductive cross-algorithm combinations.
  • Diminishing Returns via EUI: In alternating blocks, feature tuning often reaches saturation faster than hyperparameter tuning (or vice-versa). Tracking Expected Utility Improvement (EUI) naturally shifts computational resources toward the subspace that still has room for improvement.
  • Borrowing from Query Execution: Viewing black-box optimization as an operator tree provides both execution flexibility and modularity, making it easy to drop in new pipeline stages (such as pre-trained deep learning embeddings) without rewriting the core search engine.

Key Results

  • Head-to-Head Against SOTA: On the standard auto-sklearn search space, VolcanoML outperformed auto-sklearn on 25 of 30 classification tasks and 17 of 20 regression tasks. It beat TPOT on 23 of 30 classification tasks and 15 of 20 regression tasks.
  • Search Space Scalability: Evaluated across three search space sizes (Small: 20 hyperparameters, Medium: 29, Large: 100), VolcanoML's average rank advantage widened dramatically as the space grew—achieving an average rank of 1.65 on large classification tasks versus 3.57 for auto-sklearn and 3.29 for TPOT.
  • Time-to-Quality on Large Datasets: On the large Higgs dataset, VolcanoML reached a 27.2% test error in 4 hours, outperforming what auto-sklearn and TPOT achieved given a full 24-hour budget.
  • Space Enrichment:
  • Adding a custom smote_balancer operator improved balanced accuracy by up to 3.57% on the imbalanced pc2 dataset.
  • Adding an automated embedding selection stage enabled image classification on dogs-vs-cats, reaching 96.5% accuracy compared to 69.7% for baseline auto-sklearn.
  • Industrial Comparison: VolcanoML matched or outperformed commercial AutoML platforms (Google Cloud AutoML, Microsoft Azure Automated ML, Oracle Data Science, and AWS SageMaker AutoPilot) across the 6 Kaggle benchmarks under identical wall-clock limits.

Limitations and Future Work

  • Manual Plan Formulation: The paper evaluates plans selected by coarse-grained manual enumeration among five structural candidates. True automatic physical plan compilation (analogous to cost-based query optimization in RDBMSs) remains an open challenge.
  • Continuous Conditioning Variables: The conditioning block is tailored to categorical variables; continuous or high-cardinality discrete variables must currently be manually binned or discretized.
  • Independence Assumptions: Alternating blocks assume approximate conditional independence between partitioned variable subsets. When variables exhibit strong coupling, alternating optimization can converge to suboptimal local minima.

Summary updated

BibTeX
@article{li2023volcanoml,
  author = {Yang Li and Yu Shen and Wentao Zhang and Jiawei Jiang and Yaliang Li and Bolin Ding and Jingren Zhou and Zhi Yang and Wentao Wu and Ce Zhang and Bin Cui},
  title = {{VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition}},
  journal = {VLDB J.},
  year = {2023},
  doi = {10.1007/s00778-022-00752-2},
}
Figure