Resampling Methods for Uncertainty Quantification

141 concepts126 papers4 proposals

When evaluating a machine learning model or estimating a statistical parameter, a single point estimate is rarely enough. To make reliable decisions, we need to know how much that estimate fluctuates due to sampling noise. Traditional statistical inference relies on two main paths to achieve this: analytical concentration bounds (such as Hoeffding's inequality) or strict parametric assumptions (such as assuming errors follow a Gaussian distribution).

However, analytical bounds are often overly conservative, and parametric assumptions frequently fail when dealing with complex, non-linear, or high-dimensional real-world data. Resampling-based uncertainty quantification offers an elegant alternative: it uses the observed data itself as an empirical proxy for the true underlying distribution, bypassing the need for rigid theoretical assumptions.

The Core Intuition: The Data as its Own Population

To understand why resampling works, imagine an ideal scenario where we want to find the variance of an estimator. We would draw thousands of independent samples from the true population distribution, compute our estimator for each sample, and observe the spread. In reality, we only have a single dataset.

The core insight of resampling—specifically the bootstrap—is the plug-in principle: our observed sample is the best available approximation of the true population. By repeatedly sampling with replacement from this dataset, we simulate the process of drawing new samples from the universe.

True Population ──(Unknown)──> Observed Sample
                                     │
                               (Resampling)
                                     ▼
                             Bootstrap Samples ──> Empirical Distribution

As Jonny Proppe demonstrates, this empirical approach allows us to quantify model prediction uncertainty without forcing arbitrary distributional assumptions like normality onto the data. This is exceptionally valuable when dealing with non-smooth parameters or complex estimators where analytical variance formulas simply do not exist, a challenge highlighted by So et al. in the context of genomic heritability estimates.

Propagating Uncertainty Through Complex Pipelines

One of the greatest strengths of resampling is its modularity. If a pipeline contains multiple non-linear stages—such as extracting principal components and then fitting a downstream model—analytically tracking how errors propagate through the matrix operations is mathematically intractable. With bootstrapping, you simply run the entire pipeline on each resampled dataset.

For instance, Dorabiala et al. utilize this mechanism to construct confidence intervals for Principal Component Analysis (PCA), providing a built-in measure of feature reliability that widens automatically when the input data is heavily corrupted. Similarly, in topological data analysis, Monod et al. use bootstrap algorithms to estimate confidence bands for complex multiparameter persistence landscapes, enabling robust statistical inference on noisy, high-dimensional geometric structures.

Advanced Resampling for Structured Data

Simple independent resampling assumes that data points are independent and identically distributed (i.i.d.). When this assumption is violated by temporal, spatial, or structural dependencies, standard bootstrapping breaks down because it destroys the underlying correlation structure. To preserve these relationships, researchers use specialized resampling variants:

  • The Wild Bootstrap: When data exhibits heteroskedasticity (where noise levels vary across observations), the wild bootstrap perturbs the residuals of a model fit by random weights rather than resampling the data points directly. Lauzon et al. apply this technique to Diffusion Tensor Imaging, allowing them to map uncertainty across brain microarchitectures where noise varies significantly by region.
  • The Multiplier Bootstrap: To avoid explicitly estimating complex joint distributions across continuous spaces, the multiplier bootstrap scales estimators by random noise variables. Luo et al. leverage this to achieve simultaneous statistical inference across continuous state spaces in reinforcement learning, calculating critical values efficiently without knowing the system's exact dynamics.
  • The Block Bootstrap: When data has spatial or temporal dependencies, the block bootstrap samples contiguous "blocks" of data instead of individual points, preserving local correlations. As Anja Rammig demonstrates, this approach is vital when comparing large-scale ecosystem models against ground-based observations, where non-uniform spatial distributions would otherwise bias standard inference.

From Estimation to Robust Optimization

Beyond merely calculating "error bars," resampling actively guides robust decision-making. In distributionally robust optimization (DRO) and system identification, practitioners must define an "uncertainty set" around their estimates to protect against worst-case scenarios.

While analytical concentration bounds offer formal safety guarantees, they tend to be highly conservative, leading to suboptimal performance because they plan for mathematically possible but empirically highly improbable worst cases. As Matni et al. note, empirical bootstrap methods yield tighter, more practical confidence intervals for system identification.

By using bootstrap resampling to calibrate the radius of these uncertainty sets, frameworks like the one proposed by Nemmour et al. reduce over-conservatism, allowing for improved objective performance without compromising safety guarantees. This data-driven calibration is similarly applied by Aikawa et al. to establish realistic empirical boundaries for inverse covariance matrices in hierarchical time-series forecasting.

The Cost of Empirical Safety

If resampling is so versatile, why do we ever use analytical bounds? The primary bottleneck is computational. Generating an empirical distribution requires fitting a model hundreds or thousands of times.

As Klein et al. point out, relying on non-parametric bootstrapping introduces significant computational overhead, and the numerical precision of the resulting intervals is inherently limited by the number of bootstrap samples used. Furthermore, the stochastic nature of resampling introduces simulation noise. This limitation motivates researchers like Dugas et al. to derive mathematically exact bootstrap distributions for specific performance curves, aiming to combine the distribution-free benefits of bootstrapping with the speed and precision of analytical solutions.

Key Foundational Work

BOAT-Optimistic Decision Tree Construction (1999) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_resampling_based_uncertainty_quantificat_acd6f94c,
  title = {resampling-based uncertainty quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/resampling-based-uncertainty-quantification/rdr_acd6f94c62ea40bfbfebc6acb8c84544},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_resampling_based_uncertainty_quantificat_acd6f94c_v1,
  title = {resampling-based uncertainty quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/resampling-based-uncertainty-quantification/rdr_acd6f94c62ea40bfbfebc6acb8c84544/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake