Abstract
Boosting algorithms typically rely on a single type of weak learner, which limits their ability to leverage diverse domain knowledge and can lead to overfitting when "strong" learners are introduced. We present MBoost, a novel extension of AdaBoost that incorporates multiple, heterogeneous weak learners within a single ensemble. MBoost employs an internal validation step to arbitrate between models and update data weights, ensuring focus on examples that generalize poorly. Experimental results on UCI datasets demonstrate that MBoost outperforms traditional model selection via cross-validation in both accuracy and computational efficiency.
Introduction
Standard boosting algorithms like AdaBoost iteratively build a strong classifier from a sequence of simple "weak learners." However, real-world problems often benefit from combining diverse modeling approaches—such as Decision Trees, SVMs, and Naive Bayes—into a heterogeneous ensemble. Simply providing a "bag" of such models to a standard booster can introduce significant inductive bias or lead to severe overfitting, especially if one model memorizes the training data. MBoost addresses this by integrating model arbitration and validation directly into the boosting framework.
Methods
MBoost modifies the core AdaBoost loop through three primary mechanisms:
- The Arbitrator: In each round, MBoost trains a variety of weak learners from a provided "bag." It then acts as an arbitrator, selecting the single best hypothesis from this pool to add to the ensemble.
- Internal Validation: To safeguard against overfitting, MBoost splits the available data into training and validation sets at every iteration. Models are trained on the training set but evaluated on the validation set.
- Validation-Based Reweighting: Unlike AdaBoost, which updates weights based on training error, MBoost reweights the data distribution based on validation performance. This forces the algorithm to focus on examples that the models struggle to generalize on, rather than those they failed to memorize.
The algorithm naturally halts when no model in the bag can outperform random guessing on the validation set, providing an automatic stopping condition.
Key Results
The effectiveness of MBoost was evaluated on five binary classification datasets from the UCI repository: ADULT, BREAST-CANCER, CRX, HORSE-COLIC, and IONOSPHERE.
Generalization and Overfitting
MBoost demonstrates superior generalization by effectively filtering out hypotheses that overfit the training data. As shown in the comparison of training and test accuracy, MBoost maintains high performance on unseen data even as training accuracy approaches 1.0, avoiding the performance degradation often seen in standard boosting when complex learners are used.
Comparison with Cross-Validation
A common approach to model selection is to perform 10-fold cross-validation (CV) on several model types and boost the best one. MBoost proves to be a more efficient alternative. In head-to-head comparisons with "BestCV-Ada" (which uses expensive 10-fold cross-validation to select the best model type), MBoost achieved higher accuracy on three datasets and comparable results on the others, often at a fraction (often 1/10th) of the computational cost.
Conclusion
MBoost provides a robust framework for managing heterogeneous models and domain knowledge. By moving validation "inside" the boosting loop, it allows for the inclusion of strong learners without the typical overfitting penalties. The results suggest that boosting a diverse set of models together is not only more accurate but also significantly more efficient than traditional cross-validation for model selection.
Summary updated