Conversational Explanations of Machine Learning Predictions Through Class-contrastive Counterfactual Statements

Kacper Sokol, Peter A. FlachIJCAI 2018 · DOI

TL;DR

This paper tackles the "black box" problem of machine learning by proposing a conversational approach to explain predictions. Instead of generic explanations, users can interact with the model to get personalized, contrastive counterfactuals that highlight what would need to change for a different outcome, making AI more understandable and trustworthy.

What Problem Does This Paper Address?

Machine learning models are increasingly making critical decisions in areas like finance, healthcare, and the justice system. However, many of these models are proprietary and their decision-making processes are opaque, leading to a lack of trust and understanding. This is particularly problematic for non-experts who need to comprehend why a particular decision was made. Existing explanation methods often provide pre-packaged, technical details that are difficult for laypeople to grasp. The authors aim to bridge this gap by developing user-centric, interactive explanation methods that empower users to explore and understand model predictions in a way that suits their specific needs.

Methods

The core of the proposed method is a conversational, class-contrastive counterfactual explanation system. This approach is:

  • Model-agnostic: It can work with any machine learning model, not requiring specific architectures.
  • User-centric: The explainee drives the conversation, steering the explanation to their specific needs rather than receiving a fixed template.
  • Contrastive: Explanations focus on what would need to change for a different outcome. For example, "Your loan application was declined. If you had a higher income, it would have been accepted."
  • Counterfactual: These statements describe hypothetical scenarios that lead to a different prediction.
  • Dialogue-based: The interaction is framed as a dialogue, potentially through text or voice interfaces, allowing for iterative refinement of the explanation.

The authors specifically mention using decision trees as a starting point, leveraging their internal structure to measure pairwise distances between leaves. This distance metric quantifies the minimal changes in feature space required to alter a prediction, favoring sparse and concise counterfactual statements.

Datasets

The paper mentions using the UCI German credit (Statlog) dataset for demonstrating their class-contrastive counterfactual explanation for a classification outcome. They note that their approach is generalizable to other datasets with human-interpretable features, which facilitates easier evaluation of explanation quality.

Key Insights and Ideas

  • Empowering the Explainee: The most significant insight is shifting the control of the explanation from the system to the user. This allows for personalized understanding, which is crucial for non-expert users.
  • Contrastive Counterfactuals as Natural Explanations: The paper highlights that contrastive counterfactuals align well with how humans naturally explain decisions, making them more intuitive and effective.
  • Beyond Simple Decision Trees: While decision trees are used as a starting point, the authors argue that even seemingly transparent models like decision trees can become complex and difficult to explain when they grow large. Their method offers a way to provide clearer explanations even for these complex structures.
  • Addressing GDPR and Legal Requirements: The authors point out that contrastive explanations are not only user-friendly but also legally sufficient and can exceed GDPR requirements for explainability.
  • Identifying Model Flaws: The explainability approach can help uncover biases or errors within the machine learning model itself, such as how changes in income might incorrectly affect a credit score.

Key Results

The paper doesn't present extensive quantitative results in terms of accuracy metrics or performance benchmarks. Instead, the key results are qualitative and conceptual:

  • Demonstration of a Class-Contrastive Counterfactual: The paper provides a concrete example of a counterfactual explanation for a loan application scenario.
  • Methodological Framework: The primary contribution is the proposed framework for conversational, class-contrastive counterfactual explanations, which is presented as a novel and effective approach to machine learning interpretability.
  • Potential for Bias Detection: The approach is shown to have the potential to identify biases and errors in machine learning models.

Limitations and Future Work

Limitations:

  • The current work focuses on datasets with human-interpretable features. Explaining models with abstract or high-dimensional features remains a challenge.
  • While the method is model-agnostic in principle, the specific implementation for generating counterfactuals might differ across model families.

Future Work:

  • Improving Counterfactual Generation: Enhancing the distance metric to incorporate human preferences and feature actionability.
  • Story Generation: Developing algorithms to create coherent narratives from extracted counterfactuals, making explanations more accessible.
  • User Studies: Conducting rigorous user studies to evaluate the effectiveness and explainability power of their approach across different user groups.
  • Domain Adaptation: Adapting the approach to domains beyond credit scoring.
  • Handling Incomprehensible Features: Developing methods to express counterfactuals in the original feature domain or by learning high-level concepts.
  • Extending to Other Model Types: Designing and implementing algorithms for geometric and probabilistic models, and investigating data and model interpretability.

Summary updated

BibTeX
@inproceedings{sokol2018conversational,
  author = {Kacper Sokol and Peter A. Flach},
  title = {{Conversational Explanations of Machine Learning Predictions Through Class-contrastive Counterfactual Statements}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/836},
}
Figure