Introduction
In modern machine learning, practitioners often face two distinct but computationally expensive challenges: model selection and ensemble construction. Model selection involves finding the optimal hyperparameters for a single algorithm, while ensemble construction involves combining multiple models to improve predictive performance and robustness. Traditionally, these tasks are handled sequentially or independently, often requiring significant manual effort or redundant computation.
This paper introduces Sequential Model-Based Ensemble Optimization (ESMBO), a framework designed to automate the creation of high-performing ensembles during the hyperparameter search process. By leveraging the intermediate models generated during standard optimization, ESMBO builds ensembles with virtually no additional training cost.
Methodology: The Agnostic Bayes Approach
ESMBO builds upon the Agnostic Bayes framework, which treats the selection of the "best" model as a posterior distribution over the model space. To bridge the gap between searching for a single best model and building an ensemble, the authors employ two key techniques:
- Bootstrapping for Diversity: To simulate uncertainty and encourage diversity, the method creates $N$ different perspectives of the data by sampling the validation set with replacement. Each bootstrap sample represents a different "opinion" on which hyperparameter configurations are most effective.
- Round-Robin SMBO: Instead of running $N$ independent optimization processes, ESMBO runs a single process. In each iteration, it cycles through the $N$ bootstrap perspectives to suggest the next configuration to evaluate. Crucially, once a model is trained, its performance is evaluated against all $N$ bootstrap sets.
This "round-robin" update trick allows every single model trained during the search to contribute to $N$ different Gaussian Process (GP) surrogate models simultaneously. The final ensemble is then composed of the models that performed best according to each bootstrap perspective.
Experimental Setup
The authors evaluated ESMBO across a diverse set of tasks and model spaces. This included 22 regression datasets from the Louis Torgo collection and 39 classification datasets from the AYSU collection, plus MNIST. They tested the method across several popular algorithms, such as Support Vector Machines (SVM/SVR), Random Forests, Gradient Boosting, and Multilayer Perceptrons (MLP).
The performance of ESMBO was compared against standard Sequential Model-Based Optimization (SMBO), Random Search (RS), and an ensemble of models found via Random Search (ERS).
Key Results
The experimental results demonstrate that ESMBO consistently matches or exceeds the performance of its competitors across various algorithms and datasets.
As shown in the figures above, the benefits of ESMBO are particularly evident in complex hyperparameter spaces:
- Winning Frequency: In Support Vector Regression tasks, ESMBO outperformed standard SMBO in 82% of cases. In Multilayer Perceptrons, it won in 57% of cases.
- Expected Rank: ESMBO consistently achieved a lower (better) expected rank compared to SMBO and Random Search variants across iterations.
- Efficiency: Because it reuses every training run, ESMBO constructs these superior ensembles without increasing the total training time relative to a standard hyperparameter search.
Discussion and Future Work
ESMBO offers a "free" performance boost by repurposing the search history of an automated hyperparameter tuner. By optimizing for multiple bootstrap samples, the search process also becomes more robust, often avoiding local optima that might appear favorable on a single validation split.
However, some trade-offs remain. While the training cost is unchanged, the resulting ensemble increases the computational burden during inference, as multiple models must be stored and executed. Future research could explore ensemble pruning techniques to mitigate this overhead. Additionally, while this study focused on Gaussian Process surrogates, the ESMBO framework is theoretically compatible with other surrogate models like Random Forests or Tree-structured Parzen Estimators.
Summary updated