Posterior Summarization for Variable Selection in Bayesian Tree Ensembles

Shengbin Ye, Meng LiarXiv 2025 · DOI · arXiv

Abstract

Variable selection remains a fundamental challenge in statistics, especially in nonparametric settings where model complexity can obscure interpretability. Bayesian tree ensembles, particularly the popular Bayesian additive regression trees (BART) and their rich variants, offer strong predictive performance with interpretable variable importance measures. We modularize Bayesian tree ensemble methods into model choice (tree prior) and posterior summary, and provide a unified review of existing strategies, including permutation-based inference, sparsity-inducing priors, and clustering-based selection. Though typically framed as a modeling task, we show that variable selection with tree ensembles often hinges on posterior summarization, which remains underexplored. To this end, we introduce the VC-measure (Variable Count and its rank variant) with a clustering-based threshold. This posterior summary is a simple, tuning-free plug-in that requires no additional sampling beyond standard model fitting, integrates with any BART variant, and avoids the instability of the median probability model and the computational cost of permutations. In extensive experiments, it yields uniform $F_1$ gains for both general-purpose and sparsity-inducing priors; when paired with the Dirichlet Additive Regression Tree (DART), it overcomes pitfalls of the original summary and attains the best overall balance of recall, precision, and efficiency. Practical guidance on aligning summaries and downstream goals is discussed.

Introduction

Bayesian Additive Regression Trees (BART) have become a "gold standard" for nonparametric regression, prized for their ability to capture complex, nonlinear relationships without extensive manual tuning. However, like many ensemble methods, they are often viewed as a "black box." In high-dimensional settings with hundreds of potential predictors, identifying which variables actually drive the outcome is a critical but difficult task.

Existing approaches to variable selection in BART generally fall into two categories: computationally intensive permutation-based methods (like BART-VIP), which require fitting the model dozens of times to establish a noise baseline, or unstable heuristic methods (like the Median Probability Model), which can be overly sensitive to noise. This research argues that the field has focused heavily on designing better priors (model assumptions) while neglecting the importance of better posterior summarization—the process of interpreting results after the model has run.

The VC-Measure Framework

The researchers introduce the VC-measure (Variable Count and its rank variant), a "plug-and-play" posterior summary that modularizes the selection process. The method introduces three key innovations:

  1. From VIP to VC: Traditionally, researchers use Variable Inclusion Proportion (VIP), which is normalized so that all proportions sum to one. The authors demonstrate that this normalization creates a "fixed-budget" constraint where noise variables "steal" importance from true signals. The VC-measure uses raw, absolute split counts, preserving the signal's integrity.
  2. The 4D Summary Vector: To capture the behavior of variables across multiple MCMC chains, the method creates a four-dimensional profile for each predictor: its mean count, its 25th percentile count, its mean rank, and its 75th percentile rank.
  3. Tuning-Free Clustering: Instead of requiring an arbitrary threshold (e.g., 0.5), the method applies Hierarchical Agglomerative Clustering (HAC) to these 4D vectors. This automatically separates the "signal" cluster from the "noise" cluster without the need for expensive permutations.

Experimental Validation

The VC-measure was evaluated using the extended Feynman Symbolic Regression Database, a benchmark consisting of 100 nonlinear physics equations. Unlike real-world datasets, these equations provide a definitive ground truth for variable importance. The researchers tested 3,600 scenarios, varying sample sizes, signal-to-noise ratios (SNR), and the number of irrelevant "decoy" variables.

Key Findings

  • Uniform Performance Gains: The VC-measure yielded higher $F_1$ scores (balancing precision and recall) across nearly all tested scenarios compared to existing methods.
  • Computational Efficiency: When paired with the Dirichlet Additive Regression Tree (DART), the VC-measure achieved state-of-the-art accuracy with only 2–10 model replicates. This makes it approximately 5 to 50 times faster than permutation-based methods that require 50+ null model fits.
  • Fixing DART’s Over-Inclusion: DART is a popular variant designed for sparsity, but in low-noise settings, its standard summary often becomes "overconfident," picking up irrelevant variables. The VC-measure acts as a robust filter, maintaining a near-zero False Positive Rate (FPR) where other methods fail.

Conclusion and Future Work

The VC-measure demonstrates that sophisticated posterior summarization can significantly enhance the interpretability of Bayesian tree ensembles without increasing model complexity. While the method suggests running approximately 10 independent fits for maximum stability, it remains far more efficient than traditional permutation tests. Future research may extend this logic to other ensemble methods, such as Random Forests, or adapt it for multimodal data settings.

The implementation of the VC-measure is available as an R package at github.com/mattsheng/BartVC.

Summary updated

Abstract

Variable selection remains a fundamental challenge in statistics, especially in nonparametric settings where model complexity can obscure interpretability. Bayesian tree ensembles, particularly the popular Bayesian additive regression trees (BART) and their rich variants, offer strong predictive performance with interpretable variable importance measures. We modularize Bayesian tree ensemble methods into model choice (tree prior) and posterior summary, and provide a unified review of existing strategies, including permutation-based inference, sparsity-inducing priors, and clustering-based selection. Though typically framed as a modeling task, we show that variable selection with tree ensembles often hinges on posterior summarization, which remains underexplored. To this end, we introduce the VC-measure (Variable Count and its rank variant) with a clustering-based threshold. This posterior summary is a simple, tuning-free plug-in that requires no additional sampling beyond standard model fitting, integrates with any BART variant, and avoids the instability of the median probability model and the computational cost of permutations. In extensive experiments, it yields uniform $F_1$ gains for both general-purpose and sparsity-inducing priors; when paired with the Dirichlet Additive Regression Tree (DART), it overcomes pitfalls of the original summary and attains the best overall balance of recall, precision, and efficiency. Practical guidance on aligning summaries and downstream goals is discussed.
BibTeX
@misc{ye2025posterior,
  author = {Shengbin Ye and Meng Li},
  title = {{Posterior Summarization for Variable Selection in Bayesian Tree Ensembles}},
  year = {2025},
  eprint = {2509.07121},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.07121},
  howpublished = {arXiv},
}
Figure