Learning Performance Maximizing Ensembles with Explainability Guarantees

Vincent Pisztora, Jia Li

In high-stakes fields like medical diagnosis or criminal justice, machine learning models face a difficult "double bind." They must be incredibly accurate to be useful, but they must also be explainable to be trusted. Usually, this is a zero-sum game: you either pick a transparent "glass box" model (like a simple decision tree) and sacrifice accuracy, or you pick a high-performance "black box" (like a deep neural network) and sacrifice understanding.

A new paper from researchers at Pennsylvania State University, titled "Learning Performance Maximizing Ensembles with Explainability Guarantees," proposes a way to have our cake and eat it too. They introduce a framework called Ensembles with Explainability Guarantees (EEG) that intelligently delegates specific data points to either a glass box or a black box, ensuring a specific level of explainability without tanking performance.

TL;DR

The EEG framework allows users to set a "guaranteed" percentage of explainable predictions (e.g., "I need 70% of my predictions to come from a transparent model"). By ranking which observations are best suited for which model, the system maintains—and sometimes even exceeds—the accuracy of state-of-the-art black-box models.

What Problem Does This Paper Address?

The core challenge is the performance-explainability trade-off. While "complete explainability" (using a glass box for everything) is ideal for trust, it often fails to meet the rigorous accuracy requirements of real-world applications.

Existing "partial explainability" methods—which use a mix of models—often have significant drawbacks:

  1. Model-Specific: They only work with specific types of models (e.g., only linear models or only rule-based systems).
  2. Explainability Collapse: If the models are trained together, the black box often "takes over," rendering the glass box predictions uninformative.
  3. Rigidity: Most methods require you to retrain the entire system if you decide you want a different ratio of explainable-to-black-box predictions.

Methods

The EEG approach treats the problem as an optimal allocation task. Instead of training one giant hybrid model, it trains three components independently:

  1. A Glass Box ($g$): An intrinsically interpretable model (e.g., Logistic Regression).
  2. A Black Box ($b$): A high-performance model (e.g., Gradient Boosting or Neural Networks).
  3. An Allocator ($a$): A separate model that decides which data points go to which "expert."

The "secret sauce" is the Allocation Desirability Ranking. The researchers developed a ranking system that orders data points based on two criteria:

  • Sufficient Performance: Does the glass box provide a "good enough" answer here?
  • Relative Advantage: Which model is actually more accurate for this specific point?

Because the components are trained independently, the system is model-agnostic. You can swap a Neural Network for a Random Forest without changing the underlying logic. Furthermore, the ranking is "Q-Complete," meaning you can slide a dial to change the explainability level ($q$) at test time without retraining the models.

Datasets

The authors validated EEG using a benchmark suite of 31 tabular datasets (13 for classification and 18 for regression). These included standard ML benchmarks like:

  • Medical/Social: Wine quality, Credit risk, Medical charges.
  • Physical/Technical: California Housing, Electricity demand, Superconductivity.
  • Large-scale: MiniBooNE (particle physics), Jannis.

Key Insights and Ideas

  • The "Free Lunch" Effect: On average, the researchers found they could provide explainable predictions for 74% of observations without any loss in performance compared to using a black box for everything.
  • Synergy via Complementarity: In some datasets (like the "PolR" dataset), the ensemble actually outperformed both individual models. This happens when the glass box and black box have "complementary areas of expertise"—the allocator learns to send the "spiral patterns" to the black box and the "diamond patterns" to the glass box (see Figure 1 in the paper).
  • The "Kitchen Sink" Allocator: The allocator performs best when it has access not just to the raw data, but also to the predictions of both models and the "disagreement" (distance) between them. This helps it identify regions where the black box is likely correcting a glass box error.

Key Results

  • High Explainability Thresholds: The "95TQM" metric showed that, on average, users could achieve 94% explainability while still maintaining 95% of the performance of the best individual model.
  • Superiority over Random: The method captured roughly 40% more performance than a random allocation strategy (measured via PPCR).
  • Robustness: The system consistently identified "sufficiency categories"—knowing when a model was likely to fail—with an average accuracy of 76%.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Tabular Focus: The study focused entirely on tabular data. While this is where glass boxes are most common, extending this to image or text data (perhaps using "concept-based" glass boxes) remains a frontier.
  • Feature Dependency: The "feature-dependent" allocator (which looks at the data itself) didn't always provide a massive boost over a simpler "feature-independent" allocator that just looks at model disagreement. This suggests that for some datasets, the "expertise" of a model is more about global uncertainty than local feature patterns.
  • Model Diversity: The authors suggest that using a more diverse set of component models might further improve the "synergy" where the ensemble beats both parents.

By decoupling the "how to predict" from the "who should predict," EEG provides a practical, mathematically grounded path toward AI that is both high-performing and legally/ethically compliant.

Summary updated

Abstract

In this paper we propose a method for the optimal allocation of observations between an intrinsically explainable glass box model and a black box model. An optimal allocation being defined as one which, for any given explainability level (i.e. the proportion of observations for which the explainable model is the prediction function), maximizes the performance of the ensemble on the underlying task, and maximizes performance of the explainable model on the observations allocated to it, subject to the maximal ensemble performance condition. The proposed method is shown to produce such explainability optimal allocations on a benchmark suite of tabular datasets across a variety of explainable and black box model types. These learned allocations are found to consistently maintain ensemble performance at very high explainability levels (explaining $74\%$ of observations on average), and in some cases even outperforming both the component explainable and black box models while improving explainability.
BibTeX
@inproceedings{pisztora2024learning,
  author = {Vincent Pisztora and Jia Li},
  title = {{Learning Performance Maximizing Ensembles with Explainability Guarantees}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i13.29378},
}
Figure