TL;DR
This paper investigates whether machine-generated visual interpretations help everyday users understand why image classifiers make mistakes. Surprisingly, they found that showing these interpretations actually decreased users' accuracy in guessing the incorrect labels by about 10%, suggesting that current interpretation methods might not be as helpful as hoped for end-users.
What Problem Does This Paper Address?
As deep neural networks (DNNs) become more prevalent in critical applications like healthcare and transportation, their "black box" nature poses a significant challenge. While these models achieve high performance, understanding why they make certain predictions, especially errors, is crucial for building trust and improving them. Researchers have developed various methods to generate "interpretations" – visual cues that highlight which parts of an input (like an image) the model focused on. However, it's unclear if these interpretations are genuinely useful for "end users" – people who aren't AI experts but are impacted by the system's outputs – in understanding and reasoning about model failures. This paper aims to bridge this gap by directly evaluating the usefulness of these visual interpretations for general users trying to decipher incorrect predictions.
Methods
The core of this research is a human-centric evaluation using a task called "Guessing the Incorrectly Predicted Label." The authors trained an ImageNet classifier and identified images it misclassified. For each misclassified image, they presented it to online crowd workers (simulating general users) along with its correct label and the information that the model had made an error.
The experiment involved two conditions:
- Baseline (No-Int): Workers saw the image and correct label, and were asked to guess the incorrectly predicted label from a set of five options (one correct incorrect label, four distractors).
- Interpretation (Int): Workers saw the same as the baseline, plus machine-generated visual interpretations. These interpretations were generated using three common post-hoc methods:
- Input Perturbation: Showing how the model's output changes when parts of the input are "deleted" or masked.
- Inter-Feature Extraction (GradCAM): Highlighting discriminative regions using intermediate network layers.
- Output Backpropagation (SmoothGrad): Using gradients to map input pixels to the model's output.
The key hypothesis was that if interpretations effectively explained the model's reasoning, users presented with them would be better at guessing the incorrect label than those without.
Datasets
The study used images from the ImageNet dataset. The authors trained their own image classifier on ImageNet and then selected misclassified images from this trained model for their experiments. They did not introduce a new benchmark dataset but rather used existing misclassified examples from a standard dataset.
Key Insights and Ideas
- Interpretations Can Be Counterproductive: The most striking finding is that providing visual interpretations hindered users' ability to guess the incorrectly predicted label, rather than helping them. This challenges the common assumption that more information, especially visual cues, is always better for understanding.
- Specific Error Types Are Particularly Affected: The negative impact of interpretations was most pronounced for errors related to "Similar Appearance Inference" (where the model confuses visually similar objects) and "Correlation Learning" (where the model learns spurious correlations, like associating a bathroom context with a shower curtain). This suggests that current interpretations might not effectively disentangle these complex failure modes.
- Potential for Misleading Users: The results imply that interpretations, while aiming to explain, might sometimes introduce noise or lead users down the wrong path, making them less accurate than if they relied on their own intuition.
Key Results
The study conducted two experiments with a total of 3,800 submissions from 150 online crowd workers.
- Experiment 1: Accuracy in the "Interpretation" group was 0.73, compared to 0.81 in the "No-Interpretation" group. This difference was statistically significant (p<0.05).
- Experiment 2: Accuracy in the "Interpretation" group was 0.63, compared to 0.73 in the "No-Interpretation" group. This difference was also statistically significant (p<0.01).
Across both experiments, showing interpretations decreased the average guessing accuracy by approximately 10%.
Limitations and Future Work
The authors acknowledge several limitations:
- Small Sample Size: While 200 images were used, this is a small fraction of ImageNet's misclassifications, and the authors suggest more data is needed.
- Limited Interpretation Types: Only three types of interpretations were tested, and they were presented together. It's possible that other interpretation methods, or presenting them individually, might yield different results.
- Crowd Worker Noise: MTurk workers, while simulating general users, can introduce noise due to misunderstandings or lack of expertise.
- Task Specificity: The study focused solely on image classification. It's unclear if these findings generalize to other AI tasks.
Future work should explore:
- Why Interpretations Fail: Deeper investigation into the reasons behind the ineffectiveness, potentially focusing on the interpreters' representational power, format, or how they interact with specific error types.
- Better Interpretation Formats: Developing new ways to present interpretations that are more intuitive and informative for end-users.
- Human-AI Collaboration: Exploring how interpretations can be more effectively integrated into collaborative workflows where humans and AI work together.
- Generalization: Testing these findings on different tasks and with different user groups.
Summary updated