On Semi-parametric Inference for BART

Veronika RockováICML 2020

Introduction

Bayesian Additive Regression Trees (BART) have long been considered a "gold standard" for predictive modeling in machine learning. However, in fields like medicine or economics, accurate predictions are not enough; practitioners must also know how sure the model is. This requires rigorous statistical inference—constructing confidence intervals and performing hypothesis tests.

While Bayesian models like BART naturally provide "credible intervals" through their posterior distributions, a critical theoretical question has remained unanswered: do these Bayesian intervals behave like frequentist confidence intervals as the amount of data increases? This property, known as the Bernstein-von Mises (BvM) phenomenon, is well-understood for simple parametric models but notoriously difficult to prove for non-parametric models like BART, where the number of parameters can be infinite and the tree structure is learned directly from the data.

This paper provides the first rigorous proof that BART's posterior distributions for smooth functionals are asymptotically Gaussian. By doing so, it theoretically justifies the use of BART for uncertainty quantification, bridging the gap between a "black-box" predictive powerhouse and a theoretically sound tool for scientific inference.

Overcoming the Curse of Adaptivity

A major hurdle in proving BvM for non-parametric models is the "curse of adaptivity." This occurs when a model is so adept at adjusting to the smoothness of a signal that it inadvertently introduces bias into its uncertainty estimates.

To tackle this, the author focuses on semi-parametric inference. Rather than analyzing the entire regression surface simultaneously, the study examines "linear functionals"—essentially weighted averages of the predicted function. The methodology relies on several key technical components:

  • Self-Similarity Condition: To overcome the curse of adaptivity, the author employs a "self-similarity" assumption. This mathematical requirement ensures that the true underlying function does not appear "too smooth" at very small scales, helping the model correctly identify the appropriate level of complexity.
  • Hierarchical Priors: The analysis covers various configurations of BART, incorporating different priors for tree depth (such as the Galton-Watson process) and different priors for leaf values (Gaussian versus Laplace).
Binary tree structure illustrating the hierarchical nature of BART models.
  • Bernstein-von Mises Theorems: Through new mathematical proofs, the author demonstrates that the posterior distribution of these linear functionals concentrates around a normal distribution centered at an efficient estimator.

(Note: This paper is a purely theoretical work focused on mathematical proofs and statistical theory; no empirical experiments on real-world or synthetic datasets were conducted.)

Key Results and Insights

The paper establishes several foundational theorems that validate BART's inferential capabilities:

  • Asymptotic Normality: The author proves the BvM property starting from a simplified one-dimensional case with a fixed number of tree leaves (Theorem 4.1). Crucially, Theorem 5.1 (Adaptive BvM) extends this to show that even when the tree structure and complexity are unknown and learned from data, BART still achieves asymptotic normality for a single predictor ($p=1$).
  • The BART Ensemble: Theorem 5.2 extends the proof to the full BART ensemble—a sum of many trees—for the average regression surface in multiple dimensions ($p \ge 1$).
  • Rate Optimality: The results confirm that BART not only provides the correct "shape" of uncertainty but does so while maintaining the best possible (rate-optimal) estimation speed.
  • Justifying MCMC: Practically, these theoretical results mean that the common practice of using the variation in Markov Chain Monte Carlo (MCMC) draws to create uncertainty bands is now mathematically justified for BART under specific conditions.

Limitations and Future Directions

While a significant leap forward, the paper highlights a fundamental limitation regarding smoothness. Because BART relies on step functions (trees), it inherently struggles to optimally approximate functions that are smoother than "Lipschitz" (smoothness level $\alpha > 1$). Overcoming this "smoothness ceiling" may require developing smoother versions of BART that utilize different basis functions.

Furthermore, the current proofs focus strictly on linear functionals. Establishing similar BvM results for non-linear functionals or for the entire function (fully non-parametric BvM) remains a challenging open problem. Finally, for higher dimensions ($p > 1$), the results currently require the weighting function in the functional to be constant to avoid bias. Future research could explore relaxing this restriction to allow for more complex, weighted inferences.

Summary updated

BibTeX
@inproceedings{rockova2020semi,
  author = {Veronika Rocková},
  title = {{On Semi-parametric Inference for BART}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure