An Investigation of Sensitivity on Bagging Predictors: An Empirical Approach

Guohua LiangAAAI 2012 · DOI

Introduction

In real-world machine learning applications, such as fraud detection or medical diagnosis, datasets are frequently characterized by class imbalance. Standard algorithms often prioritize overall accuracy, which can lead to the neglect of the minority class. While Bagging (Bootstrap Aggregating) is a well-established technique for improving model stability and accuracy, its sensitivity to varying levels of class imbalance across different base predictors has not been extensively quantified. This study provides an empirical investigation into which bagging predictors remain robust under shifting class distributions.

Methodology

The research framework evaluates the sensitivity of 12 distinct bagging algorithms using a structured experimental design.

Experimental framework for evaluating bagging sensitivity.

The methodology involves four primary stages:

  1. Data Preparation: 14 imbalanced datasets were processed using Random Under-Sampling (RUS) to generate nine distinct levels of class distribution for each, resulting in 126 experimental variations.
  2. Evaluation: A 10-trial, 10-fold cross-validation was performed on every version of the data.
  3. Sensitivity Metric: The study introduced the "changed G-mean" ($CG$) to measure performance fluctuations. The G-mean (Geometric Mean) accounts for both minority and majority class accuracy; $CG$ quantifies the rate of change in G-mean between adjacent imbalance levels.
  4. Statistical Analysis: The Friedman test and Post-hoc Nemenyi test were employed to validate the statistical significance of the sensitivity rankings.

Key Results and Discussion

The empirical results reveal a clear hierarchy of sensitivity among bagging predictors. The study distinguishes between "insensitive" models—those maintaining a predictable performance floor—and "sensitive" models, whose performance degrades sharply as skewness increases.

Average ranks of CG performance, highlighting the spectrum from insensitive to sensitive predictors.

Robust Predictors

The most significant finding is the exceptional robustness of Multi-layer Perceptron (MLP) and Naive Bayes (NB). When bagged, these algorithms showed the least sensitivity to class distribution changes. Their performance curves remained stable even at extreme levels of imbalance, making them reliable choices for dynamic data environments.

Sensitive Predictors

Conversely, tree-based and rule-based methods such as Decision Table (DTable), RepTree, and OneR were identified as highly sensitive. Their performance fluctuated significantly across different distribution levels, suggesting they are less suitable for applications where the class balance is expected to shift over time.

Conclusion

This investigation demonstrates that not all bagging ensembles are created equal regarding class imbalance. While bagging generally improves stability, the choice of the base learner is critical. Bagging MLP and NB offer superior robustness compared to traditional tree-based bagging methods. Future research should explore the mathematical foundations of this inherent stability and extend the analysis to multi-class imbalance scenarios and alternative sampling techniques like SMOTE.

Summary updated

BibTeX
@inproceedings{liang2012investigation,
  author = {Guohua Liang},
  title = {{An Investigation of Sensitivity on Bagging Predictors: An Empirical Approach}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2012},
  doi = {10.1609/aaai.v26i1.8415},
}
Figure