Stakeholder-centric explanations for black-box decisions: an XAI process model and its application to automotive goodwill assessments

Stefan Haas, Konstantin Hegestweiler, Michael Rapp, Maximilian Muschalik, Eyke HüllermeierFrontiers Artif. Intell 2024 · DOI

TL;DR

This paper introduces a novel, iterative process model for developing explainable AI (XAI) systems that are specifically tailored to the needs of diverse stakeholders in high-stakes decision-making environments. They demonstrate its effectiveness by applying it to an automotive goodwill assessment system, showing that stakeholder involvement leads to increased trust and acceptance of AI.

What Problem Does This Paper Address?

In many critical domains like finance, healthcare, and automotive, the adoption of powerful but opaque "black-box" machine learning (ML) models is hindered by a lack of trust and understanding. While these models can offer significant benefits in terms of efficiency and accuracy, their inability to explain why they make a certain decision makes it difficult for human experts to verify, audit, and ultimately rely on them. This is particularly true in high-stakes scenarios where decisions have significant financial or personal consequences. Existing XAI research often focuses on specific aspects like algorithms or evaluation methods, but lacks a comprehensive, actionable framework for practitioners to develop XAI systems that truly meet the diverse needs of all involved parties.

Methods

The authors propose a design-science research approach, an iterative process model with six phases:

  1. Stakeholder Identification and Segmentation: Identifying all relevant interest groups (e.g., users, developers, domain experts) and categorizing them based on their roles and needs.
  2. Explanation Design: Determining which aspects of the ML model need explanation (scope: local vs. global) and selecting appropriate explanation types (e.g., feature importance, rule-based, prototypes, counterfactuals).
  3. User Interface Design: Crafting how explanations will be presented to users (e.g., text, graphics, multimedia) to ensure intelligibility and usefulness.
  4. Implementation: Integrating the chosen XAI methods and UI into the existing ML system.
  5. Technical Evaluation: Assessing the explanations' faithfulness (accuracy to the model's behavior) and stability (consistency across similar inputs).
  6. Stakeholder-Centric Evaluation: Gathering feedback from stakeholders on the explanations' understandability, usefulness, and impact on trust and task performance.

This iterative cycle emphasizes continuous feedback and refinement. For their application, they specifically focused on Kernel SHAP for generating local feature importance explanations, a model-agnostic method known for its theoretical properties.

Datasets

The study utilized real-world data from an automotive manufacturer concerning goodwill assessments. This dataset comprised tens of thousands of past goodwill requests and their outcomes. The paper does not introduce a new benchmark dataset but applies its methodology to existing, proprietary company data.

Key Insights and Ideas

  • Stakeholder-Centricity is Paramount: The most significant insight is that a one-size-fits-all approach to XAI doesn't work. Different stakeholders (e.g., assessors, IT specialists, business managers) have distinct needs and preferences for explanations.
  • Textual Explanations Reign Supreme (for this context): Surprisingly, despite the availability of graphical and multimedia options, stakeholders in this automotive use case showed a strong preference for textual explanations, particularly a "decision-logic-enhanced text" format. This suggests that natural language explanations, even if less detailed than graphical ones, can be more intuitive and actionable for non-technical users.
  • Feature Importance Wins Over Rules: Contrary to expectations that users might prefer rule-based explanations due to existing systems, stakeholders favored feature importance methods. This might stem from negative experiences with older, complex rule systems that are no longer interpretable.
  • Trust is Built Through Transparency and Involvement: The study clearly demonstrates that involving stakeholders throughout the XAI development process, and providing them with understandable explanations, significantly increases their trust in ML systems and their willingness to adopt them.
  • XAI Fosters Fairness Discussions: Beyond just trust, XAI can facilitate important discussions about fairness and potential biases in automated decision-making processes.

Key Results

  • Stakeholder Preferences: In the explanation design phase, local feature importance methods were identified as the most promising by all stakeholder groups, with a statistically significant preference from model users.
  • Preferred Representation: The "decision-logic-enhanced text" visualization was overwhelmingly preferred by stakeholders, indicating its effectiveness in conveying information clearly.
  • Increased Trust: A second survey revealed that all stakeholder groups agreed that the developed XAI system increased their trust in the decision support system and positively impacted their task performance.
  • Technical Evaluation: Kernel SHAP demonstrated good faithfulness and stability for the goodwill assessment task, with minor limitations in stability for certain aspects (labor costs). Runtime and memory consumption were manageable for asynchronous use.

Limitations and Future Work

  • Domain Specificity: The authors acknowledge that the proposed process model is a general guide and needs to be tailored to specific use cases and domains (e.g., healthcare, finance). Future work could involve providing more concrete guidance and tool support for different domains.
  • White-Box Models: The current model focuses on post-hoc explanations for black-box models. Future research should also consider scenarios where interpretability is a design goal from the outset, incorporating inherently interpretable white-box models.
  • Survey Limitations: The initial survey's purely textual format might have limited stakeholders' understanding of different explanation methods. The authors suggest that more tangible, domain-specific examples are crucial for effective evaluation.
  • Stakeholder Segmentation: Further refinement of stakeholder segmentation, particularly within the "model breaker" group, could lead to even more tailored explanations.
  • Real-time Explanations: While Kernel SHAP was suitable for asynchronous explanations, applications requiring real-time explanations might need more efficient SHAP estimators.
  • Automation Bias: The increased trust in automated systems could lead to over-reliance. Future work should explore ways to monitor decision processes and prevent unilateral decisions, fostering a true human-in-command approach.

Summary updated

BibTeX
@article{haas2024stakeholder,
  author = {Stefan Haas and Konstantin Hegestweiler and Michael Rapp and Maximilian Muschalik and Eyke Hüllermeier},
  title = {{Stakeholder-centric explanations for black-box decisions: an XAI process model and its application to automotive goodwill assessments}},
  journal = {Frontiers Artif. Intell.},
  year = {2024},
  doi = {10.3389/frai.2024.1471208},
}
Figure