Explaining the Uncertainty in AI-Assisted Decision Making

Thao LeAAAI 2023 · DOI

TL;DR

This paper explores how to explain AI model uncertainty to improve human decision-making and trust. The authors propose a novel counterfactual explanation method for model confidence and evaluate its effectiveness in user studies, finding it improves understanding and trust.

What Problem Does This Paper Address?

In AI-assisted decision-making, it's not enough for an AI to simply make a prediction; users need to understand how confident the AI is in that prediction. Prior research has shown that communicating uncertainty can help calibrate user trust. However, a key gap exists in explaining why an AI is confident or not confident. This paper tackles this challenge by investigating how to effectively explain model uncertainty to foster better human understanding, trust, and ultimately, improved decision-making when interacting with AI systems.

Methods

The core of this work is a novel approach to explaining model confidence using counterfactual explanations (CF). While CFs are commonly used to explain why a model made a specific prediction (e.g., "if this feature were different, the prediction would change"), this paper extends them to explain confidence levels. Specifically, their method generates CFs that show how to change input values to achieve a different confidence score for the same class.

They propose two forms of these explanations:

  1. Example-based counterfactuals: Presenting concrete examples of input changes and their impact on confidence.
  2. Visualisation-based counterfactuals: Using visual representations to illustrate the relationship between input features and confidence scores.

These explanations were evaluated through user studies involving 180 participants across two domains. The studies measured understanding (via task prediction), trust (using a Likert scale), and satisfaction with the explanations.

Datasets

Not discussed in the paper. The paper mentions user studies were conducted in "two different domains," but specific datasets are not detailed.

Key Insights and Ideas

  • Explaining why an AI is confident is crucial: Simply providing a confidence score isn't enough. Users benefit from understanding the drivers behind that confidence level.
  • Counterfactuals can explain confidence: The paper demonstrates that extending counterfactual explanations to address confidence scores is a viable and effective approach.
  • Both example-based and visualisation-based CFs improve trust and understanding: Compared to no explanation, both methods showed positive impacts.
  • Careful design is needed for continuous variables: When using example-based CFs for continuous variables, users might resort to case-based reasoning and overlook the underlying linear correlations.
  • Visualisations need completeness: For visualisation-based CFs, showing all relevant counterfactual points is important to prevent users from under-extrapolating correlations.

Key Results

  • Both example-based and visualisation-based counterfactual explanations significantly increased user trust and understanding compared to a baseline with no explanation.
  • There was minimal difference in the effectiveness of example-based versus visualisation-based explanations in improving understanding, trust, and satisfaction.
  • Qualitative analysis revealed that users tend to use case-based reasoning with example-based explanations, potentially missing underlying correlations.
  • Users may struggle to extrapolate correlations from incomplete visualisations, highlighting the need to present all relevant counterfactual points.

Limitations and Future Work

Limitations: * The paper notes that when not all counterfactual points are shown in visualisations, users may not extrapolate beyond the shown range. * Users might overlook linear correlations with example-based explanations for continuous variables, relying instead on case-based reasoning.

Future Work: * Further explore the differences between aleatoric (data) uncertainty and epistemic (model) uncertainty and how they impact decision-making. * Define separate explanation models for aleatoric and epistemic uncertainty. * Investigate how to communicate the uncertainty within explanations themselves to further aid decision-making. * Develop explanations that provide both positive and negative evidence to help users form their own judgments rather than solely relying on the AI. * The authors anticipate having a new decision-making model by the workshop date (February 7, 2023), indicating ongoing development.

Summary updated

BibTeX
@inproceedings{le2023explaining,
  author = {Thao Le},
  title = {{Explaining the Uncertainty in AI-Assisted Decision Making}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26920},
}
Figure