Regional Tree Regularization for Interpretability in Black Box Models

Mike Wu, Sonali Parbhoo, Michael C. Hughes, Ryan Kindle, Leo A. Celi, Maurizio Zazzi, Volker Roth, Finale Doshi-Velez

Abstract

Deep neural networks (DNNs) have become indispensable in modern AI, yet their inherent complexity often renders them "black boxes," limiting their use in safety-critical domains like healthcare. This paper proposes regional tree regularization, a technique that constrains deep models to be well-approximated by simple, human-readable decision trees within specific regions of the input space. By optimizing for regional simplicity using an $L_0$ SparseMax penalty, the authors demonstrate that models can maintain state-of-the-art accuracy while providing interpretable logic that aligns with expert domain knowledge. The approach is validated on clinical datasets, showing high fidelity and clinical relevance.

Introduction

The lack of transparency in deep learning models poses a significant barrier to trust and accountability. In fields such as medicine, understanding the rationale behind a prediction is as crucial as the prediction itself. Existing interpretability methods generally offer either global summaries, which often oversimplify complex data, or local explanations, which may lack consistency across similar inputs. Regional tree regularization addresses this gap by requiring models to be "regionally" simple—interpretable within predefined, human-intuitive contexts. This approach allows the model to adapt its logic to specific patient types or environmental conditions, mirroring the way human experts reason.

Methodology

The core of the approach is a complexity penalty added to the neural network's training objective. Instead of a single global surrogate, the model is regularized against $R$ separate decision trees, one for each predefined region.

Regional Simplicity and the $L_0$ Penalty

Interpretability is quantified using the Average Path Length (APL)—the average number of "yes/no" questions a human has to answer to reach a decision in a tree. To ensure a balanced distribution of complexity across regions, the authors employ an $L_0$ regional penalty via a differentiable SparseMax transformation. This prevents the optimizer from "cheating" by over-simplifying easy regions to allow high complexity in others, leading to a more uniform and stable level of interpretability across the entire input space.

Evolution of 2D decision boundaries under regional vs. global tree regularization.

Optimization Innovations

Training surrogate-regularized models is notoriously difficult. To stabilize the process, the authors introduced three key innovations:

  1. Data Augmentation: Synthetic model weights are generated using Dirichlet distributions to better train the surrogate estimators.
  2. Deterministic CART: Fixing random seeds in the tree-building algorithm reduces optimization noise.
  3. Tree Pruning: Redundant branches are removed to minimize complexity without sacrificing fidelity to the underlying deep model.

Datasets

The authors evaluated their method on a diverse set of benchmarks, including standard UCI datasets such as Bank, Gamma, Adult, and Wine. More importantly, the method was tested on high-stakes healthcare datasets: MIMIC-III, for predicting clinical interventions like vasopressors and sedation in intensive care units, and EUResist, for predicting HIV medication response and patient mortality. These real-world applications demonstrate the model's utility in environments where interpretability is a prerequisite for deployment.

Results and Discussion

Across all datasets, regional tree regularization found "desirable" optima—models that were significantly more accurate than global trees while maintaining a low APL (complexity).

Performance and Fidelity

Regional regularization successfully recovers complex decision boundaries that global methods miss, without the noise associated with purely local approaches. The distilled trees achieved approximately 89% fidelity to the underlying deep model's predictions, proving the explanations were faithful to the "black box" they aimed to describe.

Comparison of (a) True, (b) Global, (c) Local, and (d) Regional decision boundaries in a 1D task.

Clinical Validation

In the HIV and ICU tasks, specialist clinicians confirmed that the distilled regional trees matched their cognitive categories and medical knowledge. For example, the model correctly identified that patients with lower immunity require more aggressive drug combinations, reflecting established clinical protocols and increasing the likelihood of physician adoption.

Conclusion

Regional tree regularization provides a powerful middle ground for model interpretability. By forcing deep models to "think" in simple, regional blocks, this research enables the creation of high-performance AI systems that are auditable and trustworthy enough to work alongside human experts in the most critical fields.

Summary updated

BibTeX
@inproceedings{wu2020regional,
  author = {Mike Wu and Sonali Parbhoo and Michael C. Hughes and Ryan Kindle and Leo A. Celi and Maurizio Zazzi and Volker Roth and Finale Doshi-Velez},
  title = {{Regional Tree Regularization for Interpretability in Black Box Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.6112},
}
Figure