TL;DR
This study investigates how different styles of AI explanations (feature-based vs. example-based) affect human reliance on AI. The findings suggest that example-based explanations can lead to more appropriate reliance, but also a concerning over-reliance on incorrect AI advice, highlighting the need to tailor explanations to users and tasks rather than solely focusing on interpretability.
What Problem Does This Paper Address?
As Artificial Intelligence (AI) becomes more integrated into decision-making processes, ensuring humans rely on AI appropriately is crucial. Explainable AI (XAI) aims to achieve this by making AI decisions more understandable. However, there's conflicting evidence on whether different XAI explanation styles actually improve human-AI reliance. Some studies suggest XAI can lead to over-reliance on incorrect AI advice, while others show potential benefits. This paper seeks to clarify these conflicting findings by systematically comparing feature-based, example-based, and combined explanation styles to understand their specific impacts on human reliance and interpretability.
Methods
The researchers conducted a two-part, between-subjects experimental study with 274 participants.
- Part 1: Reliance Measurement: A Judge-Advisor System (JAS) framework was used. Participants were tasked with a bird image classification task. They first made an initial decision, then received AI advice with an explanation (or no explanation in the control group), and finally made a revised decision. Reliance was measured using the Appropriateness of Reliance (AoR) framework, specifically focusing on Relative AI Reliance (RAIR) and Relative Self-Reliance (RSR).
- Part 2: Interpretability Measurement: Participants were asked to predict AI suggestions on unseen images, with and without explanations. This part aimed to assess how well different explanation styles improved human understanding of AI behavior and their ability to predict AI outputs.
Four explanation-style conditions were tested:
- No explanation (control)
- Feature-based explanation (using SHAP)
- Example-based explanation (using ExMatchina)
- Combined feature- and example-based explanation
Datasets
The study used the Caltech-UCSD Birds-200-2011 (CUB) dataset for the bird image classification task. This dataset was chosen for its suitability for image classification and its availability of approximately 60 training images per class, which is beneficial for generating more visually similar nearest-neighbor examples.
Key Insights and Ideas
- Example-based explanations can be a double-edged sword: While they seem to help users calibrate their reliance on correct AI advice, they also increase the risk of over-reliance on incorrect AI suggestions.
- Task complexity and user ability matter: The effectiveness of explanations and reliance patterns differ significantly based on the complexity of the task and the participant's baseline performance.
- "More information" isn't always better: Combining feature- and example-based explanations didn't necessarily improve reliance beyond what individual styles offered, suggesting potential information overload.
- Interpretability doesn't automatically mean better reliance: Even when explanations are understandable (high interpretability), they don't always lead to more appropriate reliance.
Key Results
- Appropriate Reliance (RAIR): Participants using example-based or combined explanations showed significantly higher RAIR compared to those using feature-based explanations or no explanation. This suggests example-based explanations are better at helping users leverage correct AI reasoning.
- Self-Reliance (RSR): Participants using any form of explanation (feature-based, example-based, or combined) exhibited lower RSR compared to the no-explanation control group. This indicates a tendency to rely less on their own judgment when explanations are present.
- Interpretability: Explanation quality played a significant role. High-quality example-based explanations led to better predictive accuracy than low-quality ones, especially when the AI's prediction was incorrect. Feature-based explanations showed a stronger effect of AI correctness on predictive accuracy.
- Accuracy Shift: Low-ability participants showed greater improvement in accuracy on high-complexity cases with example-based explanations, while high-ability participants showed a slight worsening of accuracy with explanations, particularly on high-complexity cases.
Limitations and Future Work
- Limited XAI Methods: The study used only one method for feature-based (SHAP) and one for example-based (ExMatchina) explanations. Future work should explore a wider range of XAI methods and modalities (e.g., natural language explanations).
- Information Overload: The combined explanations might have led to information overload due to presenting similar information in multiple ways. Future research could explore ways to simplify combined explanations or use different modalities.
- Reliance Beyond Binary Decisions: The current reliance frameworks are binary. Future research should consider more complex scenarios with multiple correct answers or hierarchies of correctness, and incorporate real-world factors like risks and implications.
- Engagement and Cost-Benefit Theory: The findings support the cost-benefit theory of human-AI reliance, suggesting that user engagement with explanations is key. Future work could further explore how to design explanations that maximize engagement.
Summary updated