Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests

Lucas Mentch, Giles Hooker

Abstract

This work develops formal statistical inference procedures for machine learning ensemble methods. Ensemble methods based on bootstrapping, such as bagging and random forests, have improved the predictive accuracy of individual trees, but fail to provide a framework in which distributional results can be easily determined. Instead of aggregating full bootstrap samples, we consider predicting by averaging over trees built on subsamples of the training set and demonstrate that the resulting estimator takes the form of a U-statistic. As such, predictions for individual feature vectors are asymptotically normal, allowing for confidence intervals to accompany predictions. In practice, a subset of subsamples is used for computational speed; here our estimators take the form of incomplete U-statistics and equivalent results are derived. We further demonstrate that this setup provides a framework for testing the significance of features. Moreover, the internal estimation method we develop allows us to estimate the variance parameters and perform these inference procedures at no additional computational cost. Simulations and illustrations on a real dataset are provided.

Introduction

Random Forests (RF) have become a cornerstone of modern predictive modeling due to their high accuracy and robustness. However, they have traditionally been treated as "black boxes" in statistical practice. Unlike classical linear models, RFs lack a straightforward framework for quantifying uncertainty through confidence intervals or conducting formal hypothesis tests for feature significance.

The primary obstacle to statistical inference in RFs is the use of bootstrapping (sampling with replacement). Bootstrapping introduces complex dependencies between trees that make the mathematical derivation of the prediction's distribution extremely difficult. Consequently, practitioners have often relied on heuristic measures like "Variable Importance" scores, which lack the rigorous probabilistic backing required for formal scientific discovery.

Methods

The authors propose a fundamental shift in ensemble construction: replacing bootstrapping with subsampling. By training each tree on a smaller subsample of the data (sampled without replacement), the ensemble prediction can be mathematically framed as a U-statistic.

Algorithm 1: The Subbagging procedure used to construct the ensemble.

The U-Statistic Framework

A U-statistic is a class of estimators with well-established mathematical properties, first formalized by Hoeffding in 1948. By treating the subbagged ensemble as an incomplete, infinite-order U-statistic, the authors prove that predictions are asymptotically normal. This implies that as the dataset grows, the distribution of predictions for a given input vector $x^*$ converges to a normal distribution centered on the model's expected value.

Internal Variance Estimation

To make this framework practical, the authors developed an "internal" variance estimation method. This technique allows the model to estimate the variance of its own predictions while the forest is being constructed, requiring virtually no additional computational overhead. This variance estimate is the key component needed to build "free" confidence intervals.

Experimental Evaluation

The proposed methods were validated using both synthetic and real-world datasets:

  • Synthetic Simulations: The authors used Simple Linear Regression (SLR) and the Multivariate Adaptive Regression Splines (MARS) function to verify the theoretical properties of the estimator in controlled environments.
  • eBird Data: To demonstrate real-world utility, the model was applied to a dataset of ~400,000 observations of the Indigo Bunting. Using 23 features such as land cover and elevation, the model predicted bird abundance across different spatial and temporal contexts.

Key Results

The study yielded several significant findings:

  1. Distributional Accuracy: Simulations confirmed that the empirical distribution of subbagged predictions closely matches the predicted normal distribution.
  2. Reliable Coverage: The 95% confidence intervals achieved empirical coverage rates between 91% and 99%, demonstrating that the theoretical guarantees translate well to finite samples.
  3. Feature Significance: On the eBird dataset, the framework provided a formal p-value (approx. 0) confirming that 'month' is a statistically significant predictor for Indigo Bunting abundance, aligning with known migratory patterns.
  4. Efficiency: The internal variance estimation produced results comparable to much more computationally expensive external methods, making it suitable for large-scale applications.

Discussion and Future Work

While this work provides a rigorous foundation for inference in Random Forests, several areas for future research remain. The current framework provides inference for the expected prediction of the model, which may still be biased relative to the true underlying physical function if the base learners (trees) are misspecified. Additionally, while asymptotic normality is proven, the exact rate of convergence remains a theoretical challenge. Future work may also extend this U-statistic framework to other ensemble methods beyond decision trees, potentially unifying a broader class of machine learning algorithms under a single statistical umbrella.

Summary updated

Abstract

This work develops formal statistical inference procedures for machine learning ensemble methods. Ensemble methods based on bootstrapping, such as bagging and random forests, have improved the predictive accuracy of individual trees, but fail to provide a framework in which distributional results can be easily determined. Instead of aggregating full bootstrap samples, we consider predicting by averaging over trees built on subsamples of the training set and demonstrate that the resulting estimator takes the form of a U-statistic. As such, predictions for individual feature vectors are asymptotically normal, allowing for confidence intervals to accompany predictions. In practice, a subset of subsamples is used for computational speed; here our estimators take the form of incomplete U-statistics and equivalent results are derived. We further demonstrate that this setup provides a framework for testing the significance of features. Moreover, the internal estimation method we develop allows us to estimate the variance parameters and perform these inference procedures at no additional computational cost. Simulations and illustrations on a real dataset are provided.
BibTeX
@misc{mentch2014quantifying,
  author = {Lucas Mentch and Giles Hooker},
  title = {{Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests}},
  year = {2014},
  eprint = {1404.6473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1404.6473},
}

Loading related content...

Figure