Abstract
Many methods to explain black-box models, whether local or global, are additive. In this paper, we study global additive explanations for non-additive models, focusing on four explanation methods: partial dependence, Shapley explanations adapted to a global setting, distilled additive explanations, and gradient-based explanations. We show that different explanation methods characterize non-additive components in a black-box model's prediction function in different ways. We use the concepts of main and total effects to anchor additive explanations, and quantitatively evaluate additive and non-additive explanations. Even though distilled explanations are generally the most accurate additive explanations, non-additive explanations such as tree explanations that explicitly model non-additive components tend to be even more accurate. Despite this, our user study showed that machine learning practitioners were better able to leverage additive explanations for various tasks. These considerations should be taken into account when considering which explanation to trust and use to explain black-box models.
Introduction
Modern machine learning models often operate as "black boxes," utilizing highly non-linear functions and complex feature interactions to achieve high predictive accuracy. To make these models interpretable, researchers frequently employ additive explanations. These summaries decompose a prediction into a sum of individual feature effects, providing a clear "Feature A added X, Feature B subtracted Y" narrative. However, when models contain significant interactions or correlated features, there is no unique way to generate an additive summary. This research explores how different methods allocate these complex interactions and evaluates which approaches best serve human understanding.
Methodology
The study compares four primary classes of global additive explanations:
- Partial Dependence (PD) & Marginal Plots: Traditional statistical approaches that average out the effects of other features to isolate a single feature's impact.
- Global Shapley (gSHAP): This method aggregates local Shapley values—derived from cooperative game theory—into a global contribution plot. By plotting local values against feature values, a global trend line can be established.
- Distilled Additive Explanations: The core contribution of this work involves training a "student" model (such as Student Additive Splines - SAS or Student Bagged Additive Boosted Trees - SAT) to mimic the predictions of the black-box "teacher" model.
- gGRAD: A gradient-based approach utilizing first-order Taylor decompositions.
These methods were evaluated against non-additive "interpretable" models, such as Decision Trees (DT), to determine if explicitly modeling interactions improves human utility.
Experimental Setup
The methods were tested across synthetic functions with known additive/non-additive components and several real-world benchmarks, including:
- Bikeshare: Predicting rental demand (Regression).
- Loan Risk: Financial scoring (Regression).
- Medical & Credit Data: Pneumonia risk and FICO credit score challenges (Classification).
Key Results and Discussion
Distillation and Fidelity
The research found that distillation-based methods (SAT and SAS) consistently achieved the highest fidelity to the original black-box models. In the Bikeshare dataset, for instance, the SAT model achieved an RMSE of 0.98, significantly outperforming Partial Dependence (1.00) and gGRAD (1.25).
However, the study also highlighted the issue of correlation smearing. When features are highly correlated—such as "Temperature" and "Feels Like Temperature"—additive models often distribute importance across both, which can lead to misleading artifacts or "bumps" in the explanation plots, as seen in the gSHAP visualization for temperature effects.
The Interpretability Paradox
A critical finding of the study is the trade-off between model faithfulness and human usability. While Decision Trees are technically more "faithful" because they explicitly model interactions, they proved difficult for humans to navigate. In a user study of 50 expert practitioners:
- Pattern Recognition: 42% of users with additive SAT plots correctly identified complex "rush hour" demand patterns, whereas 0% of Decision Tree users could find the same pattern within the tree structure.
- Efficiency: Users interpreted additive models significantly faster (~11.7 minutes) than Decision Trees (~17.5 minutes).
- Ranking: 75% of SAT users correctly ranked top features, compared to 58% for Decision Tree users.
Conclusion
While additive models inherently simplify non-additive realities, they remain the most effective tool for human interpretation. Distilled additive models like SAT provide the best balance of fidelity and clarity. Future research should investigate the "sweet spot" of complexity—determining how many pairwise interactions can be introduced before an additive model loses its interpretability advantage over more complex structures like deep decision trees.
Summary updated