Variance Reduction through Bootstrap Aggregation

83 concepts76 papers3 proposals

High-capacity machine learning models, such as deep neural networks and complex decision trees, are often "unstable." An unstable learner is one where small perturbations in the training data lead to significant changes in the resulting model’s predictions. While this flexibility allows models to capture intricate patterns, it also makes them prone to overfitting noise.

Bootstrap Aggregation, commonly known as Bagging, is a fundamental ensemble technique designed to stabilize these learners. By training multiple versions of a model on different subsets of data and averaging their outputs, bagging effectively smooths out the idiosyncratic errors of individual learners.

The Mechanism of Resampling and Averaging

The core of bagging lies in two steps: bootstrapping and aggregating.

  1. Bootstrapping: Instead of training a single model on the entire dataset, researchers create multiple "new" datasets by sampling the original data with replacement. Each bootstrap sample is the same size as the original but contains a different mix of data points—some are repeated, and some are omitted.
  2. Aggregating: A base learner is trained independently on each bootstrap sample. For classification, the final prediction is typically determined by a plurality vote; for regression or density estimation, the predictions are averaged.

As Guohua Liang et al. demonstrate in An Empirical Study of Bagging Predictors for Different Learning Algorithms, this process acts as a variance-reduction mechanism. By combining the outputs of diverse models, the ensemble reduces the risk that the final system will be led astray by outliers or noise present in any single slice of the data.

Why It Works: Smoothing the Decision Surface

To understand why bagging works, consider the "bias-variance tradeoff." A simple linear model has high bias (it might miss complex patterns) but low variance (it is very stable). A neural network has low bias but high variance. Tony Plate et al. explored this in A Comparison between Neural Networks and other Statistical Techniques for Modeling the Relationship between Tobacco and Alcohol and Cancer, noting that while neural networks could uncover hidden patterns, they required stabilization to be reliable in medical contexts.

Bagging allows us to keep the low bias of a complex model while artificially lowering its variance. When we average multiple models, the components of their errors that are uncorrelated tend to cancel each other out. Dirk Ormoneit and Volker Tresp showed this effect in Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging, where bagging prevented density models from "collapsing" or overfitting to single data points.

Robustness in High-Noise and Low-Data Regimes

The stabilizing effect of bagging is particularly valuable when data is scarce or exceptionally noisy. In scientific computing, J. Nathan Kutz et al. introduced Ensemble-SINDy in Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control. They found that by using bagging to discover the "laws of physics" from data, they could identify the correct underlying equations even when the signal-to-noise ratio was extremely low—conditions where standard sparse regression would fail.

Similarly, in structural learning, Gal Elidan proposed Bagged Structural EM in Bagged Structure Learning of Bayesian Network. By bagging the model selection scores, the algorithm avoids getting stuck in local optima caused by the specific noise of a small dataset, leading to more coherent and generalizable Bayesian Network structures.

Mitigating Predictive Multiplicity and Arbitrariness

Recent research has highlighted a more subtle benefit of bagging: the reduction of "predictive multiplicity." This occurs when multiple models achieve similar accuracy but make different predictions for the same individual.

In Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification, A. Feder Cooper et al. argue that this variance is a form of "arbitrariness" that can undermine fairness. They propose hierarchical bagging to stabilize these outcomes. Furthermore, Hsiang Hsu et al. show in RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting that averaging candidate learners at each step of a boosting process can smooth out idiosyncratic decisions, leading to more consistent and reliable models in fairness-critical applications.

Summary of Applications

The versatility of bagging is evidenced by its successful application across diverse fields:

Key Foundational Work

Bagging, Boosting, and C4. 5 (1996) — contributed 1 concept elements to this direction.
A Comparison between Neural Networks and other Statistical Techniques for Modeling the Relationship between Tobacco and Alcohol and Cancer (1996) — contributed 1 concept elements to this direction.
Error Correlation and Error Reduction in Ensemble Classifiers (1996) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bagging_d1391234,
  title = {bagging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bagging/rdr_d1391234dce8486489883b45649744d6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bagging_d1391234_v1,
  title = {bagging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bagging/rdr_d1391234dce8486489883b45649744d6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake