Bagging Ensemble Selection for Regression

Quan Sun, Bernhard Pfahringer

TL;DR

This paper introduces and evaluates Bagging Ensemble Selection for Regression (BES-OOB), a novel strategy that extends successful ensemble selection techniques from classification to regression. BES-OOB demonstrates superior performance compared to standard Bagging and Stochastic Gradient Boosting, particularly when using diverse model libraries, and shows promise for efficient ensemble pruning.

What Problem Does This Paper Address?

Constructing accurate and stable predictive models is a central challenge in machine learning. Ensemble learning, which combines multiple models, often achieves better performance than individual learners. While ensemble selection (ES) has shown success in classification by intelligently choosing the best subset of models from a library, its application and effectiveness in regression settings were less explored. A key issue with standard ES is the potential for "hillclimb set overfitting," where the selection process optimizes performance on a validation set but fails to generalize to unseen data. Furthermore, the benefits of using diverse types of base learners within an ensemble, and efficient methods for reducing the size of large ensembles without sacrificing accuracy, were open questions for regression.

Methods

The core of this paper is the Bagging Ensemble Selection for Regression (BES-OOB) strategy. It builds upon the Ensemble Selection (ES) algorithm. The key innovation is the use of out-of-bag (OOB) samples from bagging for performance estimation during the ensemble selection process. Instead of a separate validation set, BES-OOB leverages the data not used in training each bagged model to guide the selection of base learners. This approach aims to mitigate overfitting and simplify the process by removing the need to tune a separate hillclimb set ratio.

The paper also explores two other aspects:

  • Diverse Model Libraries: BES-OOB is designed to accommodate different types of base learners. The study investigates whether combining diverse learners (e.g., regression trees, SVMs, M5P trees) within the library improves performance, especially as the library size grows.
  • Ensemble Pruning: To address the computational cost of large ensembles, the paper examines pruning techniques. Specifically, it compares simple averaging with two methods: the Cocktail Ensemble (CE) algorithm and stacking with the Non-Negative Least Squares (NNLS) algorithm. NNLS is proposed as a novel approach for pruning ensembles by learning optimal weights for the selected base learners, with the constraint that weights must be non-negative.

Datasets

The experiments were conducted on a collection of 42 regression datasets sourced from the UCI repository and StatLib. These datasets cover a range of complexities and characteristics, allowing for a comprehensive evaluation of the proposed methods. No new benchmarks were introduced; the focus was on evaluating existing and proposed methods on established datasets.

Key Insights and Ideas

  • OOB for Regression ES is Effective: Using out-of-bag samples for ensemble selection in regression (BES-OOB) proves to be a robust and effective strategy, overcoming some limitations of traditional ES.
  • Diversity Matters, Especially at Scale: The advantage of using a diverse library of base learners (different algorithms) becomes significantly more pronounced as the size of the model library increases.
  • NNLS for Pruning is Promising: Stacking with the Non-Negative Least Squares (NNLS) algorithm is a viable and effective method for pruning ensembles, not only reducing their size but also potentially improving predictive performance.
  • Simplicity and Generality: BES-OOB is presented as a simple yet general strategy that can be applied to both classification and regression tasks, making it a versatile tool.

Key Results

  • Superior Performance: BES-OOB, using regression trees as base learners, significantly outperforms standard Bagging and Stochastic Gradient Boosting (SGB) on the tested datasets. It also shows competitive performance against a combined strategy (BSGB).
  • Diversity Boost: When using a diverse model library, BES-OOB's performance ranking improves substantially as the library size increases, moving from third to first place in average ranking.
  • Efficient Pruning: Pruning methods (CE and NNLS) significantly reduce the final ensemble size and number of trees compared to simple averaging. Notably, BES-OOB-nnls achieved an average ensemble size reduction of 83% and a tree reduction of 82%, while maintaining competitive or even improved predictive performance on some datasets.

Limitations and Future Work

  • Theoretical Analysis: While the paper presents experimental results, a comprehensive theoretical analysis of the ES algorithm's convergence properties is left for future research.
  • Pruning Performance: While NNLS shows promise for pruning, further investigation into its impact on predictive performance across a wider range of datasets and ensemble configurations could be beneficial. The paper notes that NNLS not only prunes but can also improve performance, which is an interesting avenue for future work.
  • Base Learner Choice: The study primarily uses REPTree (a CART-like regression tree) as the base learner. Exploring the performance of BES-OOB with other types of base learners, beyond the diverse library experiment, could offer further insights.

Summary updated

Abstract

Bagging ensemble selection (BES) is a relatively new ensemble learning strategy. The strategy can be seen as an ensemble of the ensemble selection from libraries of models (ES) strategy. Previous experimental results on binary classification problems have shown that using random trees as base classifiers, BES-OOB (the most successful variant of BES) is competitive with (and in many cases, superior to) other ensemble learning strategies, for instance, the original ES algorithm, stacking with linear regression, random forests or boosting. Motivated by the promising results in classification, this paper examines the predictive performance of the BES-OOB strategy for regression problems. Our results show that the BES-OOB strategy outperforms Stochastic Gradient Boosting and Bagging when using regression trees as the base learners. Our results also suggest that the advantage of using a diverse model library becomes clear when the model library size is relatively large. We also present encouraging results indicating that the non-negative least squares algorithm is a viable approach for pruning an ensemble of ensembles.
BibTeX
@inproceedings{sun2012bagging,
  author = {Quan Sun and Bernhard Pfahringer},
  title = {{Bagging Ensemble Selection for Regression}},
  booktitle = {Australasian Conference on Artificial Intelligence},
  year = {2012},
  doi = {10.1007/978-3-642-35101-3\_59},
}

Loading related content...

Figure