Interpretation Quality Score for Measuring the Quality of interpretability methods

Yuansheng Xie, Soroush Vosoughi, Saeed Hassanpour

Abstract

As machine learning models are increasingly deployed in high-stakes domains, the need for reliable explanations of model behavior has grown. However, the field lacks a standardized metric to evaluate the quality of these explanations. This paper introduces the Interpretation Quality Score (IQS), a novel composite metric that integrates human-centric measures of plausibility, simplicity, and reproducibility. By benchmarking six popular interpretability methods across various Natural Language Processing (NLP) tasks, we demonstrate that IQS provides a rigorous framework for objective evaluation, revealing significant performance variations and highlighting the importance of simplicity in human-AI interaction.

Introduction

The explosion of "explainable AI" (XAI) has led to the development of numerous methods—such as LIME, SHAP, and Integrated Gradients—designed to reveal the logic behind model decisions. Despite their prevalence, the field currently faces a "wild west" scenario where no consensus exists on what constitutes a "good" explanation. Different methods often produce contradictory results for the same input, leaving researchers and practitioners without a formal benchmark to determine which method is most suitable for a specific application.

Comparison of different interpretation methods on a single sentence, illustrating the lack of consensus in feature attribution.

This paper addresses this gap by proposing the Interpretation Quality Score (IQS), moving beyond subjective evaluations toward a multi-dimensional, quantitative score.

The Interpretation Quality Score (IQS) Framework

The IQS is defined as a weighted sum of three fundamental pillars, allowing practitioners to tune the metric's sensitivity using hyperparameters ($\alpha_1, \alpha_2, \alpha_3$):

  1. Plausibility: This metric assesses how well a model's explanation aligns with human intuition. It is calculated using the Jaccard similarity between the features (e.g., specific words) identified as important by a human annotator and those flagged by the interpretability method.
  2. Simplicity: Grounded in cognitive psychology, this metric recognizes that humans have limited processing capacity (typically $7 \pm 2$ information chunks). Simplicity penalizes overly complex explanations, rewarding conciseness to ensure the output is digestible for the end-user.
  3. Reproducibility (Simulatability): This measures the functional utility of an explanation. If an explanation is high-quality, a human should be able to use it to accurately predict the model's output on new instances, effectively acting as a "human-in-the-loop."
Calculation of the Plausibility metric by comparing features identified by an interpretation method against human-annotated ground truth.

Experimental Methodology

The authors evaluated six feature-attribution methods: Input X Gradient, DeepLIFT, Kernel SHAP, LIME, Guided Backpropagation, and Integrated Gradients. These were tested using a RoBERTa model across three tasks from the GLUE benchmark:

  • SST2: Sentiment Analysis.
  • STSB: Semantic Textual Similarity.
  • QNLI: Question Natural Language Inference.

Human ground truth was established through crowdsourced annotations via Amazon Mechanical Turk (MTurk), where workers identified important words and attempted to reproduce model outputs based on provided explanations.

Key Insights and Results

The benchmarking process revealed several critical findings:

  • The Disagreement Problem: As shown in the introductory figure, methods frequently disagree on word importance. For instance, in a movie review, one method might highlight a word as positive while another marks it as negative.
  • Top Performers: Input X Gradient and DeepLIFT consistently achieved the highest IQS scores across tasks. These methods successfully balanced human plausibility with the simplicity required for effective communication. Input X Gradient's performance was particularly notable given its relative simplicity compared to more computationally expensive methods.
  • The Simplicity Trade-off: More complex methods like Integrated Gradients often suffered in rankings. By assigning importance values to nearly every word in a sentence, they increased the "cognitive load," resulting in lower simplicity scores and reduced human utility.
  • Task Sensitivity: The effectiveness of a method is task-dependent. For example, Kernel SHAP excelled in inference tasks (QNLI) but was less effective for sentiment analysis (SST2).

Conclusion

The Interpretation Quality Score (IQS) provides a necessary foundation for the objective benchmarking of interpretability methods. By incorporating simplicity and human plausibility, IQS ensures that explanations are not just technically accurate relative to the model, but also useful to the humans who rely on them. Future work will expand this framework beyond feature attribution to include counterfactual and example-based explanations, as well as refining the definition of "cognitive chunks" beyond individual words.

Summary updated

BibTeX
@misc{xie2022interpretation,
  author = {Yuansheng Xie and Soroush Vosoughi and Saeed Hassanpour},
  title = {{Interpretation Quality Score for Measuring the Quality of interpretability methods}},
  year = {2022},
  eprint = {2205.12254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.12254},
}
Figure