Explaining Question Answering Models through Text Generation

Veronica Latcinnik, Jonathan Berant

Abstract

Modern language models are often criticized for their "black-box" nature, where reasoning is hidden within millions of parameters. This paper introduces a novel framework that forces models to generate human-readable textual hypotheses before selecting an answer in multi-choice question-answering (MC-QA) tasks. By utilizing a differentiable generator-classifier architecture, the proposed model achieves performance parity with standard end-to-end models while providing a transparent window into its internal knowledge and failure modes.

Introduction

Large pre-trained language models (LMs) like BERT or ALBERT are incredibly proficient at answering questions, but they are notoriously opaque. In a standard "end-to-end" architecture, the reasoning happens entirely within hidden mathematical parameters. This creates two major issues: a lack of trust, as we cannot verify if the model is using statistical shortcuts, and difficulty in debugging, as it is nearly impossible to tell if a wrong answer stems from a lack of knowledge or a misunderstanding of the prompt.

Methods

The authors propose a two-part architecture designed to make the reasoning process explicit:

  1. The Generator: An LM (such as GPT-2 or XLNet) that takes the question and generates a "textual hypothesis"—a few natural language words representing the knowledge needed to answer.
  2. The Classifier: A second model that takes this hypothesis to select the final answer from the multiple-choice options.
The Generator-Classifier architecture for explainable question answering.

To train the Generator and Classifier together, the researchers used the Gumbel-Softmax straight-through estimator. This technique allows the model to treat discrete word choices as continuous during training, enabling gradient-based learning. To ensure the generated text remained human-readable, they incorporated a KLD Loss to prevent straying from natural language patterns and a Repetition Penalty to stop the model from repeating keywords to "shout" at the classifier.

Experimental Results

The model was evaluated on three distinct tasks: Synthetic Hypernym Extraction, CommonsenseQA (CSQA), and QASC (science-based questions). The latter was used to test zero-shot transfer—how well the model generalizes to new domains without extra training.

Performance comparison of explainable models against baselines on CSQA and QASC datasets.

Key results include:

  • Performance Parity: On the CSQA dataset, the best explainable model achieved 70.9% accuracy, virtually identical to the 71.0% accuracy of the standard "black-box" baseline.
  • Generalization: When tested on the QASC dataset without fine-tuning, the model reached 39.2%, actually outperforming the end-to-end baseline (38.9%).
  • Human Approval: In blind tests, human raters gave the model's explanations a "reasonableness" score of 0.74 out of 1.0, indicating the generated text was largely logical.

Discussion and Insights

The generated hypotheses revealed why the model failed in certain instances. For example, for a question about a "hostess," the model generated "dinner eat serve," showing it confused the role with that of a waitress. This level of insight is impossible in standard models.

The researchers also noted a trade-off between utility and naturalness. Models trained strictly for accuracy often produce "list-like" keywords, while those forced to produce grammatically perfect sentences perform slightly worse on the actual QA task. Furthermore, to prevent the classifier from "cheating" using its own internal weights, the authors found that a simple similarity classifier was most effective at forcing the generator to produce semantically relevant words.

Conclusion

This framework demonstrates that explainability does not have to come at the cost of performance. While the current hypotheses often resemble "bags of words," future work will focus on encouraging more complex syntactic structures and reducing the classifier's reliance on hidden reasoning.

Summary updated

Abstract

Large pre-trained language models (LMs) have been shown to perform surprisingly well when fine-tuned on tasks that require commonsense and world knowledge. However, in end-to-end architectures, it is difficult to explain what is the knowledge in the LM that allows it to make a correct prediction. In this work, we propose a model for multi-choice question answering, where a LM-based generator generates a textual hypothesis that is later used by a classifier to answer the question. The hypothesis provides a window into the information used by the fine-tuned LM that can be inspected by humans. A key challenge in this setup is how to constrain the model to generate hypotheses that are meaningful to humans. We tackle this by (a) joint training with a simple similarity classifier that encourages meaningful hypotheses, and (b) by adding loss functions that encourage natural text without repetitions. We show on several tasks that our model reaches performance that is comparable to end-to-end architectures, while producing hypotheses that elucidate the knowledge used by the LM for answering the question.
BibTeX
@misc{latcinnik2020explaining,
  author = {Veronica Latcinnik and Jonathan Berant},
  title = {{Explaining Question Answering Models through Text Generation}},
  year = {2020},
  eprint = {2004.05569},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.05569},
}
Figure