Learning Performance Maximizing Ensembles with Explainability Guarantees

Vincent Pisztora, Jia Li

TL;DR

This paper introduces "Ensembles with Explainability Guarantees" (EEG), a novel method for optimally allocating data points between an intrinsically interpretable "glass box" model and a high-performance "black box" model. EEG learns to maximize both overall ensemble performance and the explainability level, achieving high performance even at very high explainability levels.

What Problem Does This Paper Address?

In many critical domains like medical diagnosis and criminal justice, predictive models need to meet two crucial requirements: high performance and explainability. While purely "glass box" models offer inherent interpretability, they may not achieve the necessary performance across all data regions. Conversely, powerful "black box" models often lack transparency. Existing "partial explainability" approaches attempt to balance these by using an ensemble of models, where a subset of data is explained by a glass box model, and the rest by a black box. However, these methods often struggle to jointly optimize performance and explainability, can be prone to "explainability collapse" (where the black box model dominates), and are often limited to specific model types. This paper addresses the challenge of creating an ensemble that guarantees a desired level of explainability while maximizing overall performance, using a flexible, model-agnostic approach.

Methods

The core of EEG lies in its novel approach to learning an optimal allocation strategy. Instead of learning models and allocations simultaneously or sequentially in a constrained manner, EEG first independently trains the glass box and black box models globally. Then, it learns an "allocation desirability ranking" for each data point. This ranking quantifies how beneficial it is for a given data point to be explained by the glass box model.

The key innovations are:

  • Independent and Global Component Models: EEG trains the glass box and black box models independently and on the full dataset. This allows for the use of the most powerful models for each component and prevents "explainability collapse" where the black box model might subsume the glass box's role.
  • Allocation Desirability Ranking: This is a crucial concept. It ranks observations based on a combination of their performance sufficiency (whether the model's prediction is reliable) and absolute performance measures. This ranking is independent of the desired explainability level.
  • Q-Complete Allocation Optimality: The learned allocator directly uses the desirability ranking to provide an optimal allocation for any desired explainability level (q) after a single training phase. This is a significant improvement over methods that might only provide solutions for a few specific q values.

The allocator itself is defined as a function of a rescaled ranking, which prioritizes observations based on a combination of factors: performance sufficiency, explainable performance, and absolute performance differences between the glass box and black box models.

Datasets

The paper evaluates EEG on a benchmark suite of 31 tabular datasets (13 classification, 18 regression). These datasets are chosen to represent a diverse range of tabular data problems. The datasets are split into training, validation, and test sets following standard practices. The paper notes that the four largest scale datasets were omitted due to computational limitations.

Key Insights and Ideas

  • Explainability as a "Free Lunch": For many datasets, EEG demonstrates that it's possible to increase explainability significantly without sacrificing overall performance, and sometimes even improving it. This suggests that there's often untapped potential in combining models strategically.
  • Model Agnosticism is Powerful: By training component models independently, EEG can leverage state-of-the-art glass box and black box models tailored to specific tasks, making it highly adaptable.
  • Sufficiency is Key for Allocation: The paper highlights the importance of "sufficiency" metrics, which go beyond simple accuracy to assess if a prediction is reliable enough for explanation. This is crucial for making informed allocation decisions, especially when performance trade-offs are involved.
  • Feature Augmentation Benefits Allocation: Including predictions from the glass box and black box models, along with measures of their discrepancy, as features for the allocator significantly improves its performance. This suggests that the allocator benefits from direct information about the component models' behavior.

Key Results

  • Consistent Performance Improvement: EEG consistently and significantly outperforms random allocation, capturing approximately 37% of the potential performance gain over random allocation (measured by PPCR).
  • High Explainability Maintained: On average, EEG maintains ensemble performance at a very high explainability level, explaining 74% of observations while performing at least as well as the best individual model.
  • Outperforming Component Models: In some cases, the EEG ensemble even outperforms both the individual glass box and black box models, demonstrating synergistic benefits.
  • Robustness to Explainability Level: The 95TQM metric shows that allocation performance remains within 5% of the maximal individual model performance across nearly all explainability levels.
  • Accurate Sufficiency Estimation: The learned allocator accurately estimates the sufficiency category of observations, achieving an average accuracy of 76%.

Limitations and Future Work

  • Feature Set Selection: While the "kitchen sink" approach (using all available features, including component model predictions and their discrepancies) generally performed best, the paper notes that no single feature set is universally optimal across all datasets. This suggests that for resource-constrained settings, a careful selection or search over feature sets might be beneficial.
  • Component Model Synergy: The study on component model selection found that while combined performance often leads to better ensembles, the difference in performance between selecting models for individual superiority versus combined superiority was not statistically significant on average. This suggests that for the tested model types, there wasn't a strong indication of high relative expertise in different data regions. The authors suggest that using a more diverse set of component models could be beneficial.
  • Future Work: The paper implies that exploring more diverse component model combinations and further refining feature selection strategies for the allocator could lead to even greater improvements. The flexibility of the EEG framework also opens doors for applications in various domains where both performance and explainability are critical.

Summary updated

Abstract

In this paper we propose a method for the optimal allocation of observations between an intrinsically explainable glass box model and a black box model. An optimal allocation being defined as one which, for any given explainability level (i.e. the proportion of observations for which the explainable model is the prediction function), maximizes the performance of the ensemble on the underlying task, and maximizes performance of the explainable model on the observations allocated to it, subject to the maximal ensemble performance condition. The proposed method is shown to produce such explainability optimal allocations on a benchmark suite of tabular datasets across a variety of explainable and black box model types. These learned allocations are found to consistently maintain ensemble performance at very high explainability levels (explaining $74\%$ of observations on average), and in some cases even outperforming both the component explainable and black box models while improving explainability.
BibTeX
@inproceedings{pisztora2024learning,
  author = {Vincent Pisztora and Jia Li},
  title = {{Learning Performance Maximizing Ensembles with Explainability Guarantees}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i13.29378},
}
Figure