TL;DR
This paper investigates whether machine-generated visual interpretations help "end users" understand why image classifiers make mistakes. Surprisingly, they found that providing these interpretations actually decreased users' accuracy in guessing the incorrect labels, suggesting current interpretation methods may not be as helpful as hoped for this user group.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are increasingly used in critical domains like healthcare and transportation, but their complex, "black-box" nature makes it hard to understand why they make certain predictions, especially when they err. This lack of interpretability is a significant challenge for building trust and debugging systems. Researchers have developed methods to automatically generate "interpretations" (like saliency maps) to highlight which parts of an input an AI focused on. While these methods are valuable for researchers and engineers, their actual usefulness for general users—who are most affected by system outputs but lack technical expertise—in understanding and reasoning about AI errors remains unclear. This paper specifically tackles the question: can these machine-generated visual interpretations help ordinary users identify and understand why an image classifier made a wrong prediction?
Methods
The authors designed a human-based evaluation task using Amazon Mechanical Turk (MTurk) workers to simulate "end users." The core task involved presenting workers with an image, its correct label, and the information that a DNN had misclassified it. Workers were then asked to guess the incorrectly predicted label from a set of five options.
The study employed two experimental conditions:
- Baseline Condition (No-Int): Workers saw only the image and the correct label, with the information that it was misclassified.
- Interpretation Condition (Int): Workers saw the same information as the baseline, plus machine-generated visual interpretations.
The interpretations were generated using three common post-hoc methods:
- Input Perturbation: Showing how the output changes when parts of the input are masked (using different percentages of preserved pixels).
- Intermediate Feature Extraction: Using methods like GradCAM to highlight discriminative regions based on intermediate network layers.
- Output Backpropagation: Using methods like SmoothGrad to track information from the output back to the input.
The researchers then compared the accuracy of workers in the "Int" group versus the "No-Int" group in guessing the misclassified label. They also categorized common error types to see if interpretations helped more with specific kinds of mistakes.
Datasets
The study used images from the ImageNet dataset. The authors trained their own image classifier on ImageNet, achieving a Top-1 accuracy of 78.67%. They then randomly selected misclassified images from this classifier for their experiments. No new benchmark datasets were introduced.
Key Insights and Ideas
- Interpretations Can Hinder, Not Help: The most striking finding is that providing visual interpretations decreased users' accuracy in guessing the incorrectly predicted label. This challenges the common assumption that more information, especially visual cues, is always better for understanding.
- The "Why" Matters for Errors: The paper highlights that understanding why an AI fails is crucial for users. Interpretations are hypothesized to help users reason about the model's behavior, but the results suggest this hypothesis doesn't hold true in practice for this task.
- Specific Error Types are More Susceptible: The negative impact of interpretations was particularly pronounced for errors related to "Similar Appearance Inference" (C3) and "Correlation Learning" (C4), where models might confuse visually similar objects or rely on contextual cues. This suggests that current interpretation methods might be misleading or insufficient for these complex error scenarios.
- MTurk Workers as Proxies for General Users: The study's use of MTurk workers is a pragmatic approach to simulating the experience of non-expert users, providing insights into how these interpretations might fare in real-world applications.
Key Results
- Experiment 1: In the first experiment, the "Int" group had an accuracy of 0.73, while the "No-Int" group achieved 0.81. This difference was statistically significant (p<0.05).
- Experiment 2: The second, more strictly controlled experiment, yielded similar results. The "Int" group's accuracy was 0.63, compared to 0.73 for the "No-Int" group. This difference was also statistically significant (p<0.01).
- Overall: Across both experiments, showing visual interpretations decreased average guessing accuracy by approximately 10%.
- Qualitative: Interpretations significantly lowered human accuracy for error types C3 (Similar Appearance Inference) and C4 (Correlation Learning). Accuracy for C1 (Local Character Inference) and C2 (Multiple Objects Selection) errors slightly increased but not significantly.
Limitations and Future Work
The authors acknowledge several limitations:
- Sample Size: While 200 images were used, this is a small fraction of the total misclassifications in ImageNet.
- Interpretation Variety: Only three types of interpretations were tested, and they were presented together. It's possible that other interpretation methods, or presenting them individually, might yield different results.
- Worker Noise: MTurk workers, while qualified, can still introduce noise due to misunderstanding or lack of domain knowledge.
- Task Specificity: The study focused solely on visual interpretations for image classifiers. The findings might not generalize to other AI tasks or interpretation modalities.
Future work should explore:
- Why Interpretations Failed: Deeper investigation into why interpretations were ineffective or even detrimental is needed. Hypotheses include the interpreters not being "good enough" (insufficient representational power), the format being insufficient (saliency maps not conveying information effectively), or interpreters working poorly for specific failure cases.
- Alternative Interpretation Methods: Exploring different types of interpretations or ways to present them.
- Generalizability: Testing these findings on other AI tasks and user groups.
Summary updated