A Novel Test for Additivity in Supervised Ensemble Learners

Lucas Mentch, Giles Hooker

Abstract

Additive models remain popular statistical tools due to their ease of interpretation and as a result, hypothesis tests for additivity have been developed to assess the appropriateness of these models. However, as data grows in size and complexity, learning algorithms continue to gain popularity due to their exceptional predictive performance. Due to the black-box nature of these learning methods, the increase in predictive power is assumed to come at the cost of interpretability and inference. However, recent work suggests that many popular learning techniques, such as bagged trees and random forests, have desirable asymptotic properties which allow for formal statistical inference when base learners are built with proper subsamples. This work extends hypothesis tests previously developed and demonstrates that by enforcing a grid structure on an appropriate test set, we may perform formal hypothesis tests for additivity among features. We develop notions of total and partial additivity and demonstrate that both tests can be carried out at no additional computational cost. We also suggest a new testing procedure based on random projections that allows for testing on larger grids, even when the grid size is larger than that of the training set. Simulations and demonstrations on real data are provided.

Introduction

In modern data science, a persistent tension exists between model interpretability and predictive accuracy. While simple models like linear regression offer clear statistical inference through p-values and interaction terms, they often lack the capacity to capture complex patterns in high-dimensional data. Conversely, ensemble learners like Random Forests provide superior predictive performance but are frequently criticized as "black boxes" that lack formal inferential frameworks.

Historically, researchers seeking to identify statistically significant interactions within a Random Forest were forced to revert to simpler, less accurate models for hypothesis testing. This paper bridges this gap by introducing a formal statistical framework to test for variable interactions and "additivity" within supervised ensemble learners, leveraging their asymptotic properties to provide rigorous p-values without sacrificing predictive power.

Methodology

The proposed framework relies on recent theoretical findings demonstrating that Random Forest predictions follow a Gaussian distribution when trees are constructed using subsampling (without replacement) rather than traditional bootstrapping. This allows the ensemble to be treated as an infinite-order U-statistic.

Grid-Based Testing

The primary innovation involves defining a structured grid over the features of interest. By evaluating the model's predictions across this grid, a "difference matrix" ($D$) can be used to isolate interaction effects, analogous to classical ANOVA techniques.

Visualization of a function $F(x_1, x_2)$ evaluated over a structured grid in the feature space.

The authors define two types of additivity:

  1. Total Additivity: The response is a pure sum of individual feature effects.
  2. Partial Additivity: Interactions may exist between specific subsets of variables but not others.

Random Projections for High-Dimensional Grids

To address the computational challenges of estimating large covariance matrices for dense grids, the authors introduce a random projection method. By projecting high-dimensional grid predictions into lower-dimensional spaces and averaging p-values via a Bates distribution, the test maintains statistical power even when the grid size exceeds the training set size.

Experimental Results

The methodology was validated through both simulations and real-world applications.

Simulation Studies

Using various linear and non-linear functions (e.g., sinusoids, exponentials), the authors tested for Type I error and statistical power. The Random Forest additivity test performed comparably to the t-test in linear scenarios but significantly outperformed traditional methods in complex, non-linear settings. The tests also proved robust to "nuisance" variables, maintaining sensitivity to true signals amidst noise.

Case Study: Wood Thrush Population

Applying the test to ecological data from the Cornell University Lab of Ornithology, the researchers investigated the impact of pollutants (mercury, acid deposition) and environmental factors (elevation, soil pH) on bird populations. The test formally rejected the null hypothesis of additivity, identifying a significant interaction between mercury and acid deposition. This provides statistical evidence for the ecological theory that these pollutants amplify each other's negative effects.

Discussion and Conclusion

A key advantage of this approach is that the variance estimates required for testing can be calculated during the initial construction of the Random Forest, providing "inference for free" with minimal additional computational overhead. However, the validity of the test depends on careful grid selection; the authors recommend using interior quantiles to avoid edge effects and overfitting common in tree-based models.

Future work may explore automated grid optimization and the use of local data density weighting to improve performance in sparse regions of the feature space. By providing a rigorous way to look inside the "black box," this framework enhances the utility of ensemble learners for scientific discovery and formal hypothesis testing.

Summary updated

Abstract

Additive models remain popular statistical tools due to their ease of interpretation and as a result, hypothesis tests for additivity have been developed to assess the appropriateness of these models. However, as data grows in size and complexity, learning algorithms continue to gain popularity due to their exceptional predictive performance. Due to the black-box nature of these learning methods, the increase in predictive power is assumed to come at the cost of interpretability and inference. However, recent work suggests that many popular learning techniques, such as bagged trees and random forests, have desirable asymptotic properties which allow for formal statistical inference when base learners are built with proper subsamples. This work extends hypothesis tests previously developed and demonstrates that by enforcing a grid structure on an appropriate test set, we may perform formal hypothesis tests for additivity among features. We develop notions of total and partial additivity and demonstrate that both tests can be carried out at no additional computational cost. We also suggest a new testing procedure based on random projections that allows for testing on larger grids, even when the grid size is larger than that of the training set. Simulations and demonstrations on real data are provided.
BibTeX
@article{mentch2016novel,
  author = {Lucas Mentch and Giles Hooker},
  title = {{A Novel Test for Additivity in Supervised Ensemble Learners}},
  journal = {Journal of Computational and Graphical Statistics},
  year = {2016},
  doi = {10.1080/10618600.2016.1256817},
}
Figure