TL;DR
This paper investigates whether human-provided rationales, which highlight text segments supporting a classification, can improve the quality of machine-generated explanations. They found that a model trained with human rationales produced explanations that human judges preferred over those from a model using unsupervised attention. This suggests that incorporating human insights into the training process can lead to more understandable and trustworthy AI explanations.
What Problem Does This Paper Address?
The field of Explainable AI (XAI) is rapidly growing, driven by the increasing use of machine learning in critical decision-making systems. A key challenge in XAI is making AI's reasoning transparent to humans. Simultaneously, research in "learning with rationales" has shown that providing human-annotated rationales can improve a machine learning model's predictive accuracy. However, these two areas have remained largely disconnected. This paper bridges this gap by asking: can the human-provided rationales that improve model accuracy also be used to improve the quality of the machine's explanations themselves, as judged by humans?
Methods
The authors compare two variations of a Convolutional Neural Network (CNN) for text classification:
- RA-CNN (Rationale-Augmented CNN): This model is trained using both document labels and human-annotated rationales. The rationales are used to weight sentences, effectively teaching the model which parts of the text are most important for classification.
- AT-CNN (Attention-based CNN): This model uses a standard attention mechanism. The attention weights are learned without explicit supervision from human rationales, instead relying on a learned context vector to determine sentence importance.
At test time, both models generate explanations by selecting the sentences with the highest weights (either supervised rationale weights for RA-CNN or attention weights for AT-CNN). These explanations are then presented to human judges for evaluation.
Datasets
The study uses the movie review dataset from Zaidan et al. (2007). This dataset contains 2,000 reviews (1,000 positive, 1,000 negative), with 900 reviews per class annotated with human rationales. The authors use 1,800 annotated documents for training and the remaining 200 unannotated documents for testing.
Key Insights and Ideas
- Human Rationales Enhance Explanations: The core finding is that training a model with human rationales leads to explanations that are judged as better by human evaluators, even when compared to a model using unsupervised attention. This suggests that human guidance during training can directly translate to more interpretable outputs.
- Attention is Not Always Explanation: The paper touches upon recent criticisms of attention mechanisms as explanations. While AT-CNN's attention provides some level of explanation (outperforming a random baseline), RA-CNN's supervised approach yields superior results, implying that unsupervised attention might not fully capture what makes an explanation useful to humans.
- Human Evaluation is Crucial: The authors strongly advocate for human evaluation of explanations, arguing that automated metrics, while useful, don't fully capture the goal of making AI understandable to users. They acknowledge the difficulty and cost of human evaluation but deem it essential.
Key Results
- Classification Accuracy: RA-CNN achieved a classification accuracy of 90.00%, AT-CNN 88.50%, and a baseline Doc-CNN 86.00% on the test set. This confirms the finding that rationales can improve predictive accuracy.
- Human Evaluation of Explanations: In a head-to-head comparison on Amazon Mechanical Turk, human judges preferred RA-CNN's explanations 43.47% of the time, while AT-CNN's explanations were preferred 20.48% of the time. In 36.14% of cases, the explanations were deemed equally good.
- AT-CNN vs. Random Baseline: AT-CNN's explanations were chosen as better than random explanations 57.23% of the time, demonstrating that attention mechanisms, even without explicit human supervision, can provide helpful insights.
Limitations and Future Work
- Small-Scale Analysis on Misclassifications: The authors note a preliminary, small-scale analysis where RA-CNN produced explanations supporting misclassifications, while AT-CNN sometimes provided explanations that supported the correct classification despite the model's error. This raises an interesting question for future work: should explanations always support the model's decision, or could they sometimes be used to question it?
- Overlap Between Models: While RA-CNN's explanations are generally preferred, the study found that the top three explanatory sentences from RA-CNN and AT-CNN only overlapped significantly in a minority of cases. This suggests that the models learn different aspects of what constitutes a good explanation.
- Future Evaluation Avenues: The paper suggests future work could involve comparing explanations from models trained on different datasets or exploring how explanations differ when models make incorrect predictions.
Summary updated