Introduction
Bayesian Additive Regression Trees (BART) have become a cornerstone of non-parametric regression in statistics due to their exceptional predictive performance. However, a significant limitation has emerged: BART models often "hallucinate" interactions between variables that do not exist in the underlying data generating process. This "anti-conservative" behavior makes BART unreliable for structure discovery and variable selection, where identifying true interactions is as critical as prediction accuracy. This paper introduces Dirichlet Process Forests (DP-Forests), a novel ensemble method designed to provide reliable interaction detection through structural regularization.
Methodology: Dirichlet Process Forests
The core innovation of DP-Forests is the transition from an ensemble of "generalist" trees to a collection of "specialist" clusters. Standard BART allows any tree to split on any variable, leading to a "messy soup" of interactions. In contrast, DP-Forests employ a Dirichlet Process (DP) prior to group trees into clusters.
As illustrated in Figure 2, trees within the same cluster (e.g., $Z_1=1$ and $Z_2=1$) are encouraged to split on the same small subset of variables. This is achieved through a sparse "splitting proportion" vector for each cluster. By forcing trees to "pick a lane," the model imposes a natural penalty on complexity. To represent a new interaction, the model must either initiate a new cluster or add a variable to an existing sparse cluster, effectively filtering out spurious noise.
Experimental Evaluation
The authors evaluated DP-Forests against several state-of-the-art competitors, including standard BART, Sparse BART (SBART), Iterative Random Forests (iRF), and Hierarchical Group Lasso (HL-LS). Testing was conducted on several simulated benchmarks, such as the Radchenko-James and Vo-Pati models, as well as the real-world Boston Housing dataset.
Key Results
DP-Forests demonstrated superior performance in identifying the true underlying structure of the data without sacrificing predictive accuracy.
As shown in Figure 1, DP-Forests (DPF) accurately recover the true interaction graph, whereas competing methods like SBART and iRF introduce several false positive edges (hallucinations). In the "No Interaction" variant of the Vo-Pati test, DP-Forests maintained near-zero false positives, while standard BART variants detected an average of 1.5 to 2 spurious interactions.
Furthermore, DP-Forests maintained Root Mean Square Error (RMSE) comparable to or better than standard BART, proving that interpretability does not require a performance trade-off. On the Boston Housing dataset, the model identified a stable interaction between distance to employment centers (DIS) and socioeconomic status (LSTAT), providing clearer insights than competing methods.
Conclusion
DP-Forests address the hallucination problem in Bayesian tree ensembles by using Dirichlet Process clustering to enforce structural sparsity. While the method introduces some computational overhead and relies on a sparsity hyperparameter $\alpha$, it offers a significantly more reliable framework for scientific discovery. Future work remains to establish formal selection consistency proofs for this class of models.
Summary updated