Explanations for CommonsenseQA: New Dataset and Models

Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, Dinesh GargACL 2021 · DOI

TL;DR

This paper introduces ECQA, a new dataset of human-annotated explanations for CommonsenseQA, and proposes novel models for retrieving and generating these explanations. The work addresses the critical need for explainable AI in common-sense reasoning tasks, moving beyond simple answer prediction to understanding why an answer is correct.

What Problem Does This Paper Address?

Automated question answering (QA) has seen remarkable progress, often matching or exceeding human performance. However, most of this progress has focused on predicting the correct answer, leaving the crucial aspect of explanation largely unaddressed. For common-sense QA, understanding why an answer is correct is particularly important, as it requires reasoning about implicit knowledge. Existing datasets for explanation generation are scarce, and there's a lack of clear desiderata for what constitutes a good explanation. This paper tackles this gap by defining what makes a good explanation for common-sense QA and creating a dataset and models to achieve it.

Methods

The authors propose a comprehensive approach to generating explanations, encompassing both retrieval and generation.

  • Explanation Desiderata: They define an explanation as a set of positive properties (justifying the correct answer) and negative properties (refuting incorrect answers), aiming for completeness, minimality, and coherence.
  • Property Retrieval (XR): This system uses a BERT-based encoder to represent questions and properties in a latent space. It then ranks candidate properties from a corpus and selects the most relevant ones.
  • Property Generation (XGP): This model leverages a two-step fine-tuning process of GPT-2 to generate common-sense properties. The first step trains GPT-2 to produce property-like sentences, and the second step trains it to generate a set of properties for a given QA tuple.
  • Free-Flow Explanation Generation (XGF): This component uses GPT-2 to generate a coherent, natural language paragraph that synthesizes the properties into a human-readable explanation. They propose two variants, XGF-I (direct generation) and XGF-II (two-step generation using properties).

Datasets

  • CommonsenseQA (CQA): The original dataset used for the QA task.
  • ECQA (Explanations for CommonsenseQA): A newly introduced dataset created by human annotators for 11K QA pairs from CQA. It includes positive and negative properties, as well as free-flow explanations.
  • Open Mind Common Sense (OMCS): A large corpus of common-sense facts used as a "silver corpus" for property retrieval.

Key Insights and Ideas

  • The Importance of Refutation: Simply justifying the correct answer isn't enough; a good explanation must also actively refute incorrect choices. This is a key differentiator of their approach.
  • Structured vs. Free-Flow Explanations: The paper highlights the value of both structured property sets (for machine understanding and evaluation) and free-flow natural language explanations (for human readability).
  • Leveraging Pre-trained Models: The work effectively adapts powerful pre-trained language models like BERT and GPT-2 for the novel tasks of property retrieval and generation, demonstrating their versatility beyond standard QA.
  • Human-Centric Evaluation: The reliance on human validation experiments to assess explanation quality underscores the subjective nature of "good" explanations and the need for human judgment in evaluating AI systems.

Key Results

  • ECQA Dataset: A comprehensive dataset of explanations, enabling research in this area.
  • Property Retrieval (XR): The proposed XR system, using its property ranker and top-k selector, significantly outperforms the BM25 baseline, achieving a relative gain of 100% in F1 score on the gold corpus.
  • Property Generation (XGP): The XGP model achieves an F1 score of 36.4 on the STS-BERT metric, outperforming a baseline GPT-2 model (XGP-W) and the best retrieval model.
  • Free-Flow Explanation Generation (XGF): The XGF models achieve a semantic similarity score of 61.9 (STS-BERT) and 32.1 (SPICE), demonstrating their ability to generate human-like explanations. Human evaluations show ECQA explanations are preferred over prior work (CoS explanations) across comprehensiveness, refutation completeness, minimality, and overall quality.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that guaranteeing the "comprehensiveness" of annotated property sets is difficult.
  • Approximately 2% of (q,a,c) tuples were dropped due to difficulties in matching properties, and 32 questions were marked as ambiguous.
  • The free-flow explanation generation models still have room for improvement, as indicated by the semantic similarity scores.
  • Future Work:
    • Explore RL-based schemes for joint training of property ranker and selector in XR.
  • Investigate joint training of XGP and XGF-II for improved free-flow explanation generation.
  • Enhance the accuracy and interpretability of models using the ECQA dataset.

Summary updated

BibTeX
@inproceedings{aggarwal2021explanations,
  author = {Shourya Aggarwal and Divyanshu Mandowara and Vishwajeet Agrawal and Dinesh Khandelwal and Parag Singla and Dinesh Garg},
  title = {{Explanations for CommonsenseQA: New Dataset and Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.238},
}
Figure