High-capacity machine learning models, such as deep neural networks and complex decision trees, are often "unstable." An unstable learner is one where small perturbations in the training data lead to significant changes in the resulting model’s predictions. While this flexibility allows models to capture intricate patterns, it also makes them prone to overfitting noise.
Bootstrap Aggregation, commonly known as Bagging, is a fundamental ensemble technique designed to stabilize these learners. By training multiple versions of a model on different subsets of data and averaging their outputs, bagging effectively smooths out the idiosyncratic errors of individual learners.
The Mechanism of Resampling and Averaging
The core of bagging lies in two steps: bootstrapping and aggregating.
- Bootstrapping: Instead of training a single model on the entire dataset, researchers create multiple "new" datasets by sampling the original data with replacement. Each bootstrap sample is the same size as the original but contains a different mix of data points—some are repeated, and some are omitted.
- Aggregating: A base learner is trained independently on each bootstrap sample. For classification, the final prediction is typically determined by a plurality vote; for regression or density estimation, the predictions are averaged.
As Guohua Liang et al. demonstrate in An Empirical Study of Bagging Predictors for Different Learning Algorithms, this process acts as a variance-reduction mechanism. By combining the outputs of diverse models, the ensemble reduces the risk that the final system will be led astray by outliers or noise present in any single slice of the data.
Why It Works: Smoothing the Decision Surface
To understand why bagging works, consider the "bias-variance tradeoff." A simple linear model has high bias (it might miss complex patterns) but low variance (it is very stable). A neural network has low bias but high variance. Tony Plate et al. explored this in A Comparison between Neural Networks and other Statistical Techniques for Modeling the Relationship between Tobacco and Alcohol and Cancer, noting that while neural networks could uncover hidden patterns, they required stabilization to be reliable in medical contexts.
Bagging allows us to keep the low bias of a complex model while artificially lowering its variance. When we average multiple models, the components of their errors that are uncorrelated tend to cancel each other out. Dirk Ormoneit and Volker Tresp showed this effect in Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging, where bagging prevented density models from "collapsing" or overfitting to single data points.
Robustness in High-Noise and Low-Data Regimes
The stabilizing effect of bagging is particularly valuable when data is scarce or exceptionally noisy. In scientific computing, J. Nathan Kutz et al. introduced Ensemble-SINDy in Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control. They found that by using bagging to discover the "laws of physics" from data, they could identify the correct underlying equations even when the signal-to-noise ratio was extremely low—conditions where standard sparse regression would fail.
Similarly, in structural learning, Gal Elidan proposed Bagged Structural EM in Bagged Structure Learning of Bayesian Network. By bagging the model selection scores, the algorithm avoids getting stuck in local optima caused by the specific noise of a small dataset, leading to more coherent and generalizable Bayesian Network structures.
Mitigating Predictive Multiplicity and Arbitrariness
Recent research has highlighted a more subtle benefit of bagging: the reduction of "predictive multiplicity." This occurs when multiple models achieve similar accuracy but make different predictions for the same individual.
In Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification, A. Feder Cooper et al. argue that this variance is a form of "arbitrariness" that can undermine fairness. They propose hierarchical bagging to stabilize these outcomes. Furthermore, Hsiang Hsu et al. show in RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting that averaging candidate learners at each step of a boosting process can smooth out idiosyncratic decisions, leading to more consistent and reliable models in fairness-critical applications.
Summary of Applications
The versatility of bagging is evidenced by its successful application across diverse fields:
- Medical Diagnosis: Improving the stability of Alzheimer’s stage identification in Bagging Ensembles for the Diagnosis and Prognostication of Alzheimer's Disease.
- Robotics: Reducing joint torque residuals in robotic arms by over 80% through ensemble-based error compensation, as described by Yong Tao et al. in Robot hybrid inverse dynamics model compensation method based on the BLL residual prediction algorithm.
- Natural Language Processing: Enhancing the robustness of Chinese phrase structure parsing by integrating multiple TreeBanks through bagging, as shown by Meishan Zhang et al. in Multiple TreeBanks Integration for Chinese Phrase Structure Grammar Parsing Using Bagging.
Key Foundational Work
Bagging, Boosting, and C4. 5 (1996) — contributed 1 concept elements to this direction.
A Comparison between Neural Networks and other Statistical Techniques for Modeling the Relationship between Tobacco and Alcohol and Cancer (1996) — contributed 1 concept elements to this direction.
Error Correlation and Error Reduction in Ensemble Classifiers (1996) — contributed 1 concept elements to this direction.