Abstract
Machine learning algorithms aim at minimizing the number of false decisions and increasing the accuracy of predictions. However, the high predictive power of advanced algorithms comes at the costs of transparency. State-of-the-art methods, such as neural networks and ensemble methods, often result in highly complex models that offer little transparency. We propose shallow model trees as a way to combine simple and highly transparent predictive models for higher predictive power without losing the transparency of the original models. We present a novel split criterion for model trees that allows for significantly higher predictive power than state-of-the-art model trees while maintaining the same level of simplicity. This novel approach finds split points which allow the underlying simple models to make better predictions on the corresponding data. In addition, we introduce multiple mechanisms to increase the transparency of the resulting trees.
Introduction: The Accuracy-Transparency Trade-off
In modern machine learning, there is a persistent tension between accuracy and transparency. Advanced models like deep neural networks or large ensembles (e.g., XGBoost, Random Forests) achieve high predictive power but operate as "black boxes," making them difficult to audit or explain. Conversely, simple models like linear regression are highly transparent but often fail to capture complex, non-linear relationships in data.
Model trees attempt to bridge this gap by partitioning the data space into regions and fitting a simple model to each leaf. However, traditional model trees often rely on split criteria (like Mean Squared Error or Information Gain) designed for standard decision trees with constant-value leaves. These criteria do not account for how well a linear model will perform in the resulting branches, often leading to deep, complex trees that sacrifice transparency for accuracy.
Methodology: MT-GR and the Gradient-Based Split
To address these limitations, the authors introduce MT-GR (Model Tree with Gradient and Renormalization). This approach features two primary innovations:
- Gradient-Based Split Criterion: Rather than training a new model for every potential split point—a computationally prohibitive task—the algorithm utilizes a Taylor approximation of the loss function. By analyzing the gradients of the parent node's model, it mathematically estimates which split will most effectively reduce error for child models. Essentially, the gradient indicates "how much the current model wants to change" for each data point; grouping points with similar gradients leads to superior local models.
- Renormalization: Gradients are sensitive to data scale. The authors introduced a renormalization technique to adjust data within potential splits, ensuring the process is translation-invariant and that local models are trained on well-scaled data.
As illustrated in the figure above, standard cross-entropy splits (b) may fail to capture the optimal partitioning for a model-based approach. In contrast, the gradient-based split (c) identifies a more effective boundary that allows the underlying models to better fit the data distribution.
Transparency and Interpretability
A key strength of MT-GR is its ability to maintain high accuracy using very shallow trees (typically depth 1 to 3). This ensures that the resulting rules remain human-interpretable.
The figure above demonstrates a "model stump" (depth 1). A simple rule ($X27 \le 0.5$) partitions the data, and the resulting bar charts clearly show how the importance of various features (e.g., $X34$, $X46$) shifts between the two regions. This level of clarity is vital for high-impact domains like finance and medicine, where transparency is often a regulatory requirement.
Key Results
The researchers evaluated MT-GR across eight diverse public datasets, including bankruptcy prediction, medical diagnosis, and real estate.
- Superior Accuracy: MT-GR consistently outperformed standard model trees (MT-DT) across nearly all tests.
- Outperforming Black Boxes: On several datasets, such as Breast Cancer and Olympics, shallow MT-GR models (depth 3) actually outperformed XGBoost, a state-of-the-art ensemble method, while remaining significantly easier to explain.
- Efficiency: By using gradients to approximate split gain, MT-GR avoids the "brute force" approach of training thousands of models during tree construction, making it highly efficient.
Conclusion and Future Work
MT-GR demonstrates that shallow model trees can rival the performance of complex "black-box" models without sacrificing transparency. Future research directions include extending the framework to non-linear leaf models and exploring "mimic learning," where transparent model trees are used to approximate and explain the behavior of pre-existing complex models.
Summary updated