TL;DR
Random Forests are powerful "black box" models that are notoriously difficult to interpret. This paper proposes using small ensembles of "representative trees"—created by clustering the original forest—to provide a compact, interpretable surrogate that maintains high fidelity to the original model's predictions.
What Problem Does This Paper Address?
Random Forests (RFs) are a staple of modern machine learning due to their robustness and accuracy. However, an RF typically consists of hundreds or thousands of decision trees, making it impossible for a human to "read" the model's logic.
To solve this, researchers often use surrogate models: a single, simple decision tree that tries to mimic the RF. The problem is that a single tree is often an oversimplification; it might miss secondary patterns or complex interactions that the full forest captured. This paper investigates whether we can find a "middle ground"—a small ensemble of representative trees (usually 2 to 10) that is still human-readable but far more accurate than a single tree.
Methods
The authors compare two main types of representative ensembles:
- Most Representative Tree Ensembles (MRTE): The algorithm clusters the trees in the RF based on similarity and selects the most "central" existing tree from each cluster.
- Artificial Representative Tree Ensembles (ARTE): Instead of picking an existing tree, the algorithm constructs a new "artificial" tree for each cluster designed to represent that cluster's specific logic.
The Secret Sauce: Clustering by Similarity To group the trees, the authors use a metric called Weighted Splitting Variables (WSV). This doesn't just look at which variables a tree uses, but where it uses them. A split at the root of the tree is weighted more heavily than a split near the leaves, as it has a larger impact on the final prediction. They then use k-means clustering on these distances to partition the forest.
Datasets
The study evaluates these methods across two distinct environments:
- Simulated Data: Linear and non-linear scenarios designed to test how well the surrogates handle noise variables and correlated features.
- Benchmark Datasets: 19 real-world regression datasets from the OpenML AutoML Benchmark Collection, including well-known sets like Boston Housing, Diamonds, and Abalone. These datasets vary significantly in size (from hundreds to tens of thousands of observations) and feature complexity.
Key Insights and Ideas
- Clustering vs. Randomness: One of the paper's core questions was whether similarity-based clustering actually helps. The results show that it does: clustering trees by their structural similarity (WSV) generally leads to higher "fidelity" (how closely the surrogate matches the RF) than simply picking trees at random.
- The Complexity Trade-off: There is a clear "Aha!" moment regarding the difference between MRTE and ARTE. MRTEs (selected trees) tend to be more accurate and have higher fidelity. However, ARTEs (constructed trees) are significantly "slimmer"—they have far fewer terminal nodes (leaves).
- Noise Filtering: Artificial trees (ARTs) are better at ignoring "noise" variables. Because they are constructed to represent the average behavior of a cluster, the random noise that might appear in an individual tree gets filtered out.
Key Results
- Fidelity and Size: As you increase the number of trees in the surrogate ensemble (from $k=1$ to $k=10$), fidelity to the original RF and overall prediction accuracy improve significantly.
- Compactness: On benchmark datasets, ARTEs achieved lower Bayesian Information Criterion (BIC) scores than MRTEs. This means that for the level of accuracy they provide, they are much less complex, making them the better choice when human interpretability is the primary goal.
- Stability: Small ensembles are more stable than single trees. The predictions of a 5-tree ensemble vary much less across different training runs than those of a single representative tree.
- Computational Cost: There is a price for the "artificial" approach. Constructing ARTEs (median time ~12 minutes) is significantly slower than selecting MRTs (median time < 1 second), primarily due to the variable selection steps (like the Boruta algorithm) used during construction.
Limitations and Future Work
The authors identify several areas for improvement:
- The "Boruta" Bottleneck: On very large datasets, the variable selection process for artificial trees can exceed reasonable time limits (the authors set a 15-hour cap). Future work could explore faster alternatives like the "Vita" method.
- Structural Stability: While the predictions of these ensembles are stable, the authors note that the structure of the trees (the specific splits) might still change between runs. Assessing "structural stability" is a key next step.
- User Studies: Ultimately, "interpretability" is subjective. The authors suggest that future research should involve user studies to confirm that these small ensembles actually help humans understand RF models better in practice.
Summary updated