Variable Grouping Based Bayesian Additive Regression Tree

Yuhao Su, Jie Ding

Abstract

Using ensemble methods for regression has been a large success in obtaining high-accuracy prediction. Examples are Bagging, Random forest, Boosting, BART (Bayesian additive regression tree), and their variants. In this paper, we propose a new perspective named variable grouping to enhance the predictive performance. The main idea is to seek for potential grouping of variables in such way that there is no nonlinear interaction term between variables of different groups. Given a sum-of-learner model, each learner will only be responsible for one group of variables, which would be more efficient in modeling nonlinear interactions. We propose a two-stage method named variable grouping based Bayesian additive regression tree (GBART) with a well-developed python package gbart available. The first stage is to search for potential interactions and an appropriate grouping of variables. The second stage is to build a final model based on the discovered groups. Experiments on synthetic and real data show that the proposed method can perform significantly better than classical approaches.

Introduction: The Generality-Efficiency Trade-off

In modern regression tasks, practitioners often face a fundamental trade-off between model generality and statistical efficiency. Standard ensemble methods, such as Random Forests (RF) or Bayesian Additive Regression Trees (BART), are highly general and capable of modeling complex interactions between any set of variables. However, these methods often ignore the underlying additive structure inherent in many real-world functions.

If a target function is a sum of independent components—for instance, $f(x_1, x_2) + f(x_3, x_4)$—standard tree-based models may still attempt to find interactions between unrelated variables like $x_1$ and $x_3$. This unnecessary search space leads to higher estimation variance and slower convergence, a phenomenon often referred to as the "curse of dimensionality." Conversely, purely additive models are efficient but fail when variables do interact. The proposed Variable Grouping Based BART (GBART) bridges this gap by automatically discovering and leveraging the interaction structure of the data.

Methodology: Two-Stage GBART

The GBART framework operates in two distinct stages to optimize predictive performance:

  1. Interaction Search based Grouping (isG): This initial "detective" phase employs a greedy search algorithm guided by validation error. The goal is to identify which variables exhibit non-constant interactions. Starting from a trivial partition where all variables are in a single group, the algorithm iteratively splits variables into groups that minimize prediction error.
  2. Group-based BART: Once the groups are defined, the model constructs a "sum-of-learners" ensemble. Unlike standard BART, where any tree can split on any variable, GBART restricts each individual weak learner to variables within a single specific group.

By restricting the search space for each tree to a group of interacting variables, the model concentrates its computational resources on meaningful interactions, effectively reducing the variance of the final estimator.

Experimental Setup

The researchers evaluated GBART against standard BART and Random Forest across a variety of scenarios, including 12 synthetic datasets and real-world engineering data.

Synthetic Datasets

Twelve synthetic datasets were designed with varying dimensions ($p \in \{6, 7, 20\}$) and functional forms, including products, sums-of-squares, and trigonometric interactions. These benchmarks, including the well-known Friedman dataset, test the model's ability to recover complex nonlinear structures.

Table 1: Data generating functions for the 12 synthetic cases, showing various interaction types and dimensions.

Real-world Data

The model was also tested on the Concrete Slump Test dataset, which relates concrete ingredients (e.g., water, cement, fly ash) to flow properties. This dataset is characterized by strong interactions, particularly those involving water content, making it an ideal real-world test for variable grouping.

Key Results and Insights

GBART consistently outperformed both standard BART and Random Forest across nearly all synthetic and real-world tests.

Performance Gains

In synthetic Case 7 (a 20-variable problem), GBART achieved a Mean Squared Error (MSE) of 9.88, significantly lower than standard BART (17.15) and RF (20.43). Similar trends were observed across the majority of the 12 cases.

Table 2: Comparison of MSE (and standard errors) for GBART, BART, and RF across 12 synthetic cases. GBART shows superior performance in almost every scenario.

On the real-world concrete slump tests, GBART achieved the lowest MSE for all three output variables (slump, flow, and strength). For the slump_test1 variable, GBART recorded an MSE of 5.34, compared to 6.04 for BART and 14.97 for RF.

Statistical Efficiency

The core advantage of GBART lies in its use of structure as a prior. By preventing trees from splitting on unrelated variables, the model reduces the variance of the estimator. The results show that GBART's standard errors are generally lower than or comparable to those of BART, suggesting greater stability across different data samples.

Conclusion and Future Directions

GBART demonstrates that identifying and enforcing additive structures can significantly enhance the performance of tree-based ensembles. While the current implementation focuses on second-order interactions to maintain computational feasibility, the underlying theory supports higher-order groupings. Future work may explore more sophisticated search strategies beyond greedy algorithms and extend the GBART framework to classification and time-series modeling, where additive structures are frequently encountered.

Summary updated

Abstract

Using ensemble methods for regression has been a large success in obtaining high-accuracy prediction. Examples are Bagging, Random forest, Boosting, BART (Bayesian additive regression tree), and their variants. In this paper, we propose a new perspective named variable grouping to enhance the predictive performance. The main idea is to seek for potential grouping of variables in such way that there is no nonlinear interaction term between variables of different groups. Given a sum-of-learner model, each learner will only be responsible for one group of variables, which would be more efficient in modeling nonlinear interactions. We propose a two-stage method named variable grouping based Bayesian additive regression tree (GBART) with a well-developed python package gbart available. The first stage is to search for potential interactions and an appropriate grouping of variables. The second stage is to build a final model based on the discovered groups. Experiments on synthetic and real data show that the proposed method can perform significantly better than classical approaches.
BibTeX
@misc{su2019variable,
  author = {Yuhao Su and Jie Ding},
  title = {{Variable Grouping Based Bayesian Additive Regression Tree}},
  year = {2019},
  eprint = {1911.00922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.00922},
}
Figure