Sequential Model-Based Ensemble Optimization

Alexandre Lacoste, Hugo Larochelle, Mario Marchand, François Laviolette

Introduction

In modern machine learning, practitioners often face two distinct but computationally expensive challenges: model selection and ensemble construction. Model selection involves finding the optimal hyperparameters for a single algorithm, while ensemble construction involves combining multiple models to improve predictive performance and robustness. Traditionally, these tasks are handled sequentially or independently, often requiring significant manual effort or redundant computation.

This paper introduces Sequential Model-Based Ensemble Optimization (ESMBO), a framework designed to automate the creation of high-performing ensembles during the hyperparameter search process. By leveraging the intermediate models generated during standard optimization, ESMBO builds ensembles with virtually no additional training cost.

Methodology: The Agnostic Bayes Approach

ESMBO builds upon the Agnostic Bayes framework, which treats the selection of the "best" model as a posterior distribution over the model space. To bridge the gap between searching for a single best model and building an ensemble, the authors employ two key techniques:

  1. Bootstrapping for Diversity: To simulate uncertainty and encourage diversity, the method creates $N$ different perspectives of the data by sampling the validation set with replacement. Each bootstrap sample represents a different "opinion" on which hyperparameter configurations are most effective.
  2. Round-Robin SMBO: Instead of running $N$ independent optimization processes, ESMBO runs a single process. In each iteration, it cycles through the $N$ bootstrap perspectives to suggest the next configuration to evaluate. Crucially, once a model is trained, its performance is evaluated against all $N$ bootstrap sets.

This "round-robin" update trick allows every single model trained during the search to contribute to $N$ different Gaussian Process (GP) surrogate models simultaneously. The final ensemble is then composed of the models that performed best according to each bootstrap perspective.

Experimental Setup

The authors evaluated ESMBO across a diverse set of tasks and model spaces. This included 22 regression datasets from the Louis Torgo collection and 39 classification datasets from the AYSU collection, plus MNIST. They tested the method across several popular algorithms, such as Support Vector Machines (SVM/SVR), Random Forests, Gradient Boosting, and Multilayer Perceptrons (MLP).

The performance of ESMBO was compared against standard Sequential Model-Based Optimization (SMBO), Random Search (RS), and an ensemble of models found via Random Search (ERS).

Key Results

The experimental results demonstrate that ESMBO consistently matches or exceeds the performance of its competitors across various algorithms and datasets.

Comparison of ESMBO performance metrics (Probability of winning and Expected Rank) against baselines for SVR, GBR, and Random Forest models.

As shown in the figures above, the benefits of ESMBO are particularly evident in complex hyperparameter spaces:

  • Winning Frequency: In Support Vector Regression tasks, ESMBO outperformed standard SMBO in 82% of cases. In Multilayer Perceptrons, it won in 57% of cases.
  • Expected Rank: ESMBO consistently achieved a lower (better) expected rank compared to SMBO and Random Search variants across iterations.
  • Efficiency: Because it reuses every training run, ESMBO constructs these superior ensembles without increasing the total training time relative to a standard hyperparameter search.

Discussion and Future Work

ESMBO offers a "free" performance boost by repurposing the search history of an automated hyperparameter tuner. By optimizing for multiple bootstrap samples, the search process also becomes more robust, often avoiding local optima that might appear favorable on a single validation split.

However, some trade-offs remain. While the training cost is unchanged, the resulting ensemble increases the computational burden during inference, as multiple models must be stored and executed. Future research could explore ensemble pruning techniques to mitigate this overhead. Additionally, while this study focused on Gaussian Process surrogates, the ESMBO framework is theoretically compatible with other surrogate models like Random Forests or Tree-structured Parzen Estimators.

Summary updated

Abstract

One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO.
BibTeX
@inproceedings{lacoste2014sequential,
  author = {Alexandre Lacoste and Hugo Larochelle and Mario Marchand and François Laviolette},
  title = {{Sequential Model-Based Ensemble Optimization}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2014},
}

Loading related content...

Figure