TL;DR
Existing end-to-end AutoML systems struggle to scale when user-defined or domain-specific operations expand the search space across feature engineering, model selection, and hyperparameter tuning. Inspired by classic database query engines, VolcanoML introduces a modular framework that decomposes large AutoML search spaces into tree-structured execution plans of smaller subproblems. By combining Bayesian optimization with bandit-driven conditioning and alternating execution, VolcanoML navigates high-dimensional spaces significantly faster and more effectively than state-of-the-art tools like auto-sklearn and TPOT.
What Problem Does This Paper Address?
Building an end-to-end machine learning pipeline requires searching a massive, composite space that covers feature preprocessing, algorithm selection, and hyperparameter tuning. Most popular AutoML frameworks (such as auto-sklearn and TPOT) treat this entire pipeline as a single, joint black-box optimization problem.
In practice, users frequently need to enrich the default search space with domain-specific components—such as specialized feature normalizers, pre-trained embeddings (e.g., from TensorFlow Hub), or custom loss functions. Joint optimization over this expanded space hits an immediate scalability bottleneck: standard Bayesian optimization (BO) degrades rapidly in high dimensions. While some prior systems decomposed isolated stages (like algorithm selection from tuning), no principled abstraction existed to decompose the entire end-to-end AutoML pipeline into manageable, composable pieces.
Methods
VolcanoML models search space decomposition analogously to physical query plans in relational database management systems:
- Modular Building Blocks: Search subproblems are modeled as building blocks that expose standard interfaces (
init,do_next!,get_current_best, and expected utility queries): - Joint Block: Solves a localized subspace using Bayesian Optimization (SMAC with Expected Improvement) or multi-fidelity early stopping (MFES-HB).
- Conditioning Block: Partitions the search space by conditioning on a categorical variable (e.g., the ML algorithm). It treats each choice as an arm in a Multi-Armed Bandit (MAB) problem and prunes underperforming arms using confidence bounds on their extrapolated Expected Utility (EU).
- Alternating Block: Splits variables into two subspaces (e.g., feature engineering vs. algorithm hyperparameters) and optimizes them alternately. It uses a "rotting bandit" strategy driven by Expected Utility Improvement (EUI) to dynamically allocate search budget to whichever subspace is yielding greater marginal gains.
- Volcano-Style Execution: Execution plans form a tree of building blocks. Calling
do_next!at the root recursively invokes child nodes down to the leaves, passing current best configurations back and forth across branches. - Extensibility & Meta-Learning: The framework easily incorporates pre-trained neural feature extractors and warm-starts building blocks using meta-learning across past task runs.
[Condition on Algorithm] <-- Bandit arm elimination
/ \
[Alternating Block] [Alternating Block] <-- EUI-based switching
/ \ / \
[FE Joint] [HP Joint] [FE Joint] [HP Joint] <-- SMAC / MFES-HB
Datasets
The authors evaluated VolcanoML across multiple benchmarks:
- OpenML Repository (60 datasets):
- 40 classification tasks (30 medium datasets with 1k–12k samples, 10 large datasets with 20k–110k samples including Mnist_784, Kropt, Electricity, and Higgs).
- 20 regression tasks of medium size.
- Kaggle Datasets (6 competitions): Influence Network, Virus Prediction, Employee Access, Customer Satisfaction, Business Value, and Flavours, used to benchmark against commercial cloud AutoML platforms.
- Extensibility Benchmark: Kaggle dogs-vs-cats for multimodal/image embedding selection, along with 5 imbalanced OpenML datasets (sick, pc2, abalone, page-blocks, hypothyroid) to evaluate custom sampling operators.
Key Insights and Ideas
- Decoupling Algorithms from Features: Feature preferences vary heavily by model family, but within a given model family, feature engineering and hyperparameter tuning provide largely complementary improvements. Factoring out the algorithm choice first, followed by alternating optimization between features and hyperparameters, avoids exploring unproductive cross-algorithm combinations.
- Diminishing Returns via EUI: In alternating blocks, feature tuning often reaches saturation faster than hyperparameter tuning (or vice-versa). Tracking Expected Utility Improvement (EUI) naturally shifts computational resources toward the subspace that still has room for improvement.
- Borrowing from Query Execution: Viewing black-box optimization as an operator tree provides both execution flexibility and modularity, making it easy to drop in new pipeline stages (such as pre-trained deep learning embeddings) without rewriting the core search engine.
Key Results
- Head-to-Head Against SOTA: On the standard auto-sklearn search space, VolcanoML outperformed
auto-sklearnon 25 of 30 classification tasks and 17 of 20 regression tasks. It beatTPOTon 23 of 30 classification tasks and 15 of 20 regression tasks. - Search Space Scalability: Evaluated across three search space sizes (Small: 20 hyperparameters, Medium: 29, Large: 100), VolcanoML's average rank advantage widened dramatically as the space grew—achieving an average rank of 1.65 on large classification tasks versus 3.57 for auto-sklearn and 3.29 for TPOT.
- Time-to-Quality on Large Datasets: On the large Higgs dataset, VolcanoML reached a 27.2% test error in 4 hours, outperforming what auto-sklearn and TPOT achieved given a full 24-hour budget.
- Space Enrichment:
- Adding a custom
smote_balanceroperator improved balanced accuracy by up to 3.57% on the imbalancedpc2dataset. - Adding an automated embedding selection stage enabled image classification on dogs-vs-cats, reaching 96.5% accuracy compared to 69.7% for baseline auto-sklearn.
- Industrial Comparison: VolcanoML matched or outperformed commercial AutoML platforms (Google Cloud AutoML, Microsoft Azure Automated ML, Oracle Data Science, and AWS SageMaker AutoPilot) across the 6 Kaggle benchmarks under identical wall-clock limits.
Limitations and Future Work
- Manual Plan Formulation: The paper evaluates plans selected by coarse-grained manual enumeration among five structural candidates. True automatic physical plan compilation (analogous to cost-based query optimization in RDBMSs) remains an open challenge.
- Continuous Conditioning Variables: The conditioning block is tailored to categorical variables; continuous or high-cardinality discrete variables must currently be manually binned or discretized.
- Independence Assumptions: Alternating blocks assume approximate conditional independence between partitioned variable subsets. When variables exhibit strong coupling, alternating optimization can converge to suboptimal local minima.
Summary updated