A Human-Centered Interpretability Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé Iii, Hanna M. Wallach, Jennifer Wortman Vaughan

Abstract

Interpretability in machine learning often prioritizes mathematical axioms over human cognitive alignment. This paper introduces a framework grounded in the information-theoretic concept of Weight of Evidence (WoE) to bridge this gap. By generating contrastive, modular, and robust explanations, the framework helps users distinguish between a model's prior beliefs and the evidence provided by specific data points. We demonstrate the framework's utility through quantitative robustness tests and qualitative user studies, highlighting its ability to mitigate the base rate fallacy and manage complex multi-class classification tasks.

Introduction

Most popular interpretability methods, such as LIME or SHAP, are "math-first"—derived from axiomatic principles and only later tested for human utility. This often results in explanations that fail to align with human cognitive processes. The authors identify three major gaps in current tools:

  1. The Contrastive Gap: Humans prefer "why X instead of Y" explanations, whereas most tools only explain why a prediction is what it is.
  2. The Base Rate Fallacy: Humans often ignore prior probabilities. Current tools frequently conflate a model's "prior" (class frequency) with "evidence" (instance features).
  3. The Complexity Gap: Explaining high-dimensional multi-class predictions in a single step is overwhelming for users.

Methodology

The proposed framework leverages Weight of Evidence (WoE), which measures how much a piece of evidence $e$ supports a hypothesis $h$ against an alternative. Key technical contributions include:

  • Human-Centered Design: Explanations are designed to be contrastive, exhaustive, modular, understandable, and parsimonious.
  • Sequential WoE Meta-Algorithm: Instead of a single monolithic explanation, the tool rules out subsets of classes step-by-step. For instance, in a medical diagnosis context, it might first explain "viral vs. bacterial" and then "flu vs. other viral diseases."
  • Black-Box Estimation: For models that do not provide conditional likelihoods, the authors fit a proxy density estimator (e.g., Gaussian Naive Bayes) to the model's predictions to estimate WoE values.
Figure 2: The WoE framework enables modular, sequential explanations (e.g., Viral vs. Bacterial followed by Flu vs. Other Viral) and explicitly decomposes the posterior log-odds into prior log-odds and total weight of evidence.

Evaluation and Results

The framework was evaluated using seven standard UCI classification datasets for quantitative robustness and the Online News Popularity dataset for a qualitative user study with 10 practitioners.

  • Robustness: Across UCI datasets, WoE explanations were as stable as or more stable than LIME and SHAP, showing lower local Lipschitz constants (meaning explanations don't change wildly with slight input perturbations).
  • User Preference: Practitioners praised the sequential nature of the explanations for making complex problems manageable. They found the "ruling out" process much more intuitive than traditional heatmaps.
  • Estimation Performance: The accuracy of the WoE estimation was tested across varying input dimensions and sample sizes, demonstrating reliable performance for tabular data.
Figure 6: Quantitative analysis of Mean Squared Error and Normalized DCG Score for the WoE estimation across different input dimensions and sample sizes.

Discussion and Limitations

A key insight of this work is the mathematical disentanglement of priors from evidence. WoE separates the "prior log-odds" (base rate) from the "weight of evidence" (instance-specific features). Despite this explicit separation, the user study revealed that 7 out of 10 participants still struggled with the concept of prior class probabilities, highlighting the depth of the base rate fallacy even among experts.

Current limitations include the density estimation bottleneck, which makes the method computationally expensive for high-dimensional data like images or raw text. Future work will focus on automating feature grouping and applying the framework to more complex modalities using generative models like Normalizing Flows.

Conclusion

By grounding interpretability in information theory and human-centered design, the WoE framework provides a cognitively aligned approach to explaining machine learning models. It transforms interpretability from a static product into a sequential process, helping users navigate complex decision spaces with greater clarity.

Summary updated

BibTeX
@inproceedings{alvarezmelis2021human,
  author = {David Alvarez-Melis and Harmanpreet Kaur and Hal Daumé III and Hanna M. Wallach and Jennifer Wortman Vaughan},
  title = {{A Human-Centered Interpretability Framework Based on Weight of Evidence}},
  booktitle = {HCOMP},
  year = {2021},
  doi = {10.1609/hcomp.v9i1.18938},
}
Figure