A Gradient-Based Split Criterion for Highly Accurate and Transparent Model Trees

Klaus Broelemann, Gjergji Kasneci

Abstract

Machine learning algorithms aim at minimizing the number of false decisions and increasing the accuracy of predictions. However, the high predictive power of advanced algorithms comes at the costs of transparency. State-of-the-art methods, such as neural networks and ensemble methods, result in highly complex models with little transparency. We propose shallow model trees as a way to combine simple and highly transparent predictive models for higher predictive power without losing the transparency of the original models. We present a novel split criterion for model trees that allows for significantly higher predictive power than state-of-the-art model trees while maintaining the same level of simplicity. This novel approach finds split points which allow the underlying simple models to make better predictions on the corresponding data. In addition, we introduce multiple mechanisms to increase the transparency of the resulting trees.

The Transparency-Accuracy Trade-off

In machine learning, there is a notorious trade-off between transparency and accuracy. Complex models like deep neural networks or gradient-boosted ensembles offer high accuracy but are "black boxes," making them difficult to trust in high-stakes fields like finance or medicine. Conversely, simple models like linear regression are transparent but often lack the capacity to capture complex patterns in data.

"Model trees" attempt to bridge this gap by partitioning data into regions where simple models (like linear regression) can perform well. However, traditional methods for building these trees use split criteria—such as Mean Squared Error or Information Gain—designed for standard decision trees. These criteria look for partitions that make the data more "pure" or constant, rather than partitions that make the data easier for a model to learn. While some existing methods do optimize for leaf models, they are often computationally prohibitive because they require training new models for every potential split point.

A Novel Gradient-Based Split Criterion

To address these limitations, we propose a Gradient-Based Split Criterion that allows for efficient and effective tree construction. Instead of fully training two new models for every possible split point, our method uses a single gradient descent step and a Taylor expansion to estimate the potential reduction in loss. This allows the algorithm to evaluate many more split points quickly, making the search nearly as fast as building a standard decision tree.

Comparison of decision boundaries for Logistic Regression, traditional cross-entropy splits, and the proposed gradient-based split.

As shown in the figure above, standard model trees (b) often split data to find clusters of similar labels, which may not align with the underlying linear structure. Our gradient-based approach (c) captures "model-aware" logic, finding splits that specifically allow the leaf models to capture complex, non-linear relationships (like the V-shape in the synthetic data) that a global model (a) would miss.

Maintaining Transparency

A key advantage of this approach is the preservation of interpretability. By focusing on shallow trees (depth 1-3), the resulting model remains simple enough for human inspection. Each leaf contains a simple linear model, and the decision nodes provide clear rules for which model to apply.

Example of a shallow model tree split showing the change in feature importance (coefficients) between the left and right leaves.

In the example above, a single split on feature $X27$ creates two distinct regimes. A human user can easily see how the importance of features like $X34$ or $X46$ changes depending on the value of $X27$, providing insights that a black-box ensemble would obscure.

Key Results and Performance

We evaluated our method (MT-GR) against standard linear models, traditional decision trees (DT), state-of-the-art model trees (MT-DT), and XGBoost across several public datasets for classification and regression.

  • Superiority over Standard Model Trees: MT-GR consistently outperformed MT-DT, proving that the gradient-based criterion is better at finding useful partitions than traditional entropy or MSE-based splits.
  • Competitive with Black Boxes: In several cases, such as the Breast Cancer and Census datasets, shallow MT-GR models (depth 1-3) actually outperformed the complex XGBoost model. For instance, on the Bankruptcy dataset, a depth-3 MT-GR model achieved an AUC of 91.2, nearly matching XGBoost (93.2) while remaining fully transparent.
  • The Power of the "Stump": Even a "model stump" (a tree with only one split and two leaf models) can significantly outperform a single global linear model and often approaches the performance of much deeper trees.

Conclusion and Future Work

By leveraging a Taylor approximation of the loss function, we can build shallow model trees that rival the predictive power of complex ensembles without sacrificing transparency. Future work will focus on further enhancing "human-readability," such as adding regularization terms that encourage the algorithm to pick intuitive "round numbers" for split points, making the models even more accessible to domain experts.

Summary updated

Abstract

Machine learning algorithms aim at minimizing the number of false decisions and increasing the accuracy of predictions. However, the high predictive power of advanced algorithms comes at the costs of transparency. State-of-the-art methods, such as neural networks and ensemble methods, result in highly complex models with little transparency. We propose shallow model trees as a way to combine simple and highly transparent predictive models for higher predictive power without losing the transparency of the original models. We present a novel split criterion for model trees that allows for significantly higher predictive power than state-of-the-art model trees while maintaining the same level of simplicity. This novel approach finds split points which allow the underlying simple models to make better predictions on the corresponding data. In addition, we introduce multiple mechanisms to increase the transparency of the resulting trees.
BibTeX
@inproceedings{broelemann2019gradient,
  author = {Klaus Broelemann and Gjergji Kasneci},
  title = {{A Gradient-Based Split Criterion for Highly Accurate and Transparent Model Trees}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=ixmvUPy4Z82},
}
Figure