Clustering-Based Artificial Tree Ensembles for Random Forest Approximation

Lea L. Kronziel, Maren S. Wieder, Silke Szymczak, Björn-Hergen LaabsarXiv 2026 · arXiv

TL;DR

Random Forests are powerful "black box" models that are notoriously difficult to interpret. This paper proposes using small ensembles of "representative trees"—created by clustering the original forest—to provide a compact, interpretable surrogate that maintains high fidelity to the original model's predictions.

What Problem Does This Paper Address?

Random Forests (RFs) are a staple of modern machine learning due to their robustness and accuracy. However, an RF typically consists of hundreds or thousands of decision trees, making it impossible for a human to "read" the model's logic.

To solve this, researchers often use surrogate models: a single, simple decision tree that tries to mimic the RF. The problem is that a single tree is often an oversimplification; it might miss secondary patterns or complex interactions that the full forest captured. This paper investigates whether we can find a "middle ground"—a small ensemble of representative trees (usually 2 to 10) that is still human-readable but far more accurate than a single tree.

Methods

The authors compare two main types of representative ensembles:

  1. Most Representative Tree Ensembles (MRTE): The algorithm clusters the trees in the RF based on similarity and selects the most "central" existing tree from each cluster.
  2. Artificial Representative Tree Ensembles (ARTE): Instead of picking an existing tree, the algorithm constructs a new "artificial" tree for each cluster designed to represent that cluster's specific logic.

The Secret Sauce: Clustering by Similarity To group the trees, the authors use a metric called Weighted Splitting Variables (WSV). This doesn't just look at which variables a tree uses, but where it uses them. A split at the root of the tree is weighted more heavily than a split near the leaves, as it has a larger impact on the final prediction. They then use k-means clustering on these distances to partition the forest.

Datasets

The study evaluates these methods across two distinct environments:

  • Simulated Data: Linear and non-linear scenarios designed to test how well the surrogates handle noise variables and correlated features.
  • Benchmark Datasets: 19 real-world regression datasets from the OpenML AutoML Benchmark Collection, including well-known sets like Boston Housing, Diamonds, and Abalone. These datasets vary significantly in size (from hundreds to tens of thousands of observations) and feature complexity.

Key Insights and Ideas

  • Clustering vs. Randomness: One of the paper's core questions was whether similarity-based clustering actually helps. The results show that it does: clustering trees by their structural similarity (WSV) generally leads to higher "fidelity" (how closely the surrogate matches the RF) than simply picking trees at random.
  • The Complexity Trade-off: There is a clear "Aha!" moment regarding the difference between MRTE and ARTE. MRTEs (selected trees) tend to be more accurate and have higher fidelity. However, ARTEs (constructed trees) are significantly "slimmer"—they have far fewer terminal nodes (leaves).
  • Noise Filtering: Artificial trees (ARTs) are better at ignoring "noise" variables. Because they are constructed to represent the average behavior of a cluster, the random noise that might appear in an individual tree gets filtered out.

Key Results

  • Fidelity and Size: As you increase the number of trees in the surrogate ensemble (from $k=1$ to $k=10$), fidelity to the original RF and overall prediction accuracy improve significantly.
  • Compactness: On benchmark datasets, ARTEs achieved lower Bayesian Information Criterion (BIC) scores than MRTEs. This means that for the level of accuracy they provide, they are much less complex, making them the better choice when human interpretability is the primary goal.
  • Stability: Small ensembles are more stable than single trees. The predictions of a 5-tree ensemble vary much less across different training runs than those of a single representative tree.
  • Computational Cost: There is a price for the "artificial" approach. Constructing ARTEs (median time ~12 minutes) is significantly slower than selecting MRTs (median time < 1 second), primarily due to the variable selection steps (like the Boruta algorithm) used during construction.

Limitations and Future Work

The authors identify several areas for improvement:

  • The "Boruta" Bottleneck: On very large datasets, the variable selection process for artificial trees can exceed reasonable time limits (the authors set a 15-hour cap). Future work could explore faster alternatives like the "Vita" method.
  • Structural Stability: While the predictions of these ensembles are stable, the authors note that the structure of the trees (the specific splits) might still change between runs. Assessing "structural stability" is a key next step.
  • User Studies: Ultimately, "interpretability" is subjective. The authors suggest that future research should involve user studies to confirm that these small ensembles actually help humans understand RF models better in practice.

Summary updated

Abstract

Random Forests (RFs) offer strong prediction performance but are difficult to interpret. Most representative trees (MRTs) and artificial representative trees (ARTs) provide surrogate explanations, but a single tree may miss patterns within a forest. We investigate benefits and limitations of clustering-based representative tree ensembles for RF surrogate modeling and assess the contribution of similarity-based clustering relative to random tree partitions. Using linear and nonlinear simulations and regression benchmarks, we evaluate fidelity, prediction accuracy, complexity, and prediction stability against equally sized ensembles of top-ranked MRTs and randomly selected trees. Increasing ensemble size generally improves fidelity, accuracy, and prediction stability. ART ensembles show higher fidelity in simulations and often greater prediction stability for small ensembles. Compared with random tree partitions, similarity-based clustering generally improves ART fidelity in simulations and often on benchmarks. On benchmark datasets, MRT-based ensembles generally achieve higher fidelity and prediction accuracy, whereas ART ensembles have fewer terminal nodes. ART ensembles provide compact surrogates when reduced complexity matters and their fidelity and prediction accuracy meet application requirements. These findings clarify both the trade-offs of representative tree ensembles as RF surrogates and the contribution of similarity-based clustering beyond random clustering.
BibTeX
@misc{kronziel2026clustering,
  author = {Lea L. Kronziel and Maren S. Wieder and Silke Szymczak and Björn-Hergen Laabs},
  title = {{Clustering-Based Artificial Tree Ensembles for Random Forest Approximation}},
  year = {2026},
  eprint = {2611.18803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18803},
}
Figure