Question Answering and Question Generation as Dual Tasks

Duyu Tang, Nan Duan, Tao Qin, Ming ZhouarXiv 2017 · OpenReview

TL;DR

This paper introduces a novel framework for jointly training Question Answering (QA) and Question Generation (QG) models, treating them as "dual tasks." By leveraging the inherent probabilistic relationship between QA and QG, their proposed method improves the performance of both tasks simultaneously, achieving competitive results on standard benchmarks.

What Problem Does This Paper Address?

The paper tackles the challenge of improving Question Answering (QA) and Question Generation (QG) systems. While both tasks involve understanding the relationship between a question and an answer, they are typically trained independently. The authors propose that QA and QG are intrinsically linked – a good QA model can help a QG model generate more relevant questions, and a QG model can provide valuable signals for QA. The core problem is how to effectively exploit this "duality" to mutually enhance both tasks, rather than treating them as separate problems.

Methods

The key innovation is a joint training framework that explicitly models the probabilistic correlation between QA and QG. The authors formulate this relationship using Bayes' theorem, stating that the joint probability of a question-answer pair P(q, a) can be expressed as P(a)P(q|a) (related to QG) or P(q)P(a|q) (related to QA). Their framework aims to minimize a loss function that includes the individual losses for QA and QG, along with a regularization term that enforces this probabilistic duality.

Specifically:

  • QA Model: A discriminative recurrent neural network (RNN) based model is used for answer sentence selection. It takes a question and a candidate answer sentence and outputs a score indicating their relevance.
  • QG Model: A generative sequence-to-sequence (seq2seq) model is employed for question generation. It takes an answer sentence and generates a question.
  • Joint Training: The models are trained simultaneously. The regularization term encourages the QA model to produce probabilities P(a|q) that are consistent with the QG model's P(q|a) and the language models P(a) and P(q). This is achieved by minimizing the squared difference between the log-probabilities derived from both sides of the probabilistic duality equation.

Datasets

The paper evaluates its approach on three datasets:

  • MARCO: Originally for reading comprehension, it's adapted for answer sentence selection.
  • SQUAD: Also originally for reading comprehension, adapted for answer sentence selection.
  • WikiQA: A benchmark dataset specifically designed for answer sentence selection.

The authors note that MARCO and SQUAD are large-scale datasets, but their original task is different from answer sentence selection. They derive their answer sentence selection task by assuming that sentences containing the correct answer span are correct answers, and vice-versa.

Key Insights and Ideas

  • QA and QG as Duals: The core insight is that QA and QG are not just related but can be viewed as dual tasks, offering a powerful avenue for mutual improvement.
  • Probabilistic Duality as Regularization: Using the probabilistic correlation between QA and QG as a regularization term is an effective way to guide joint training, forcing the models to learn consistent representations.
  • End-to-End Learning: The entire framework, including both QA and QG components, is differentiable, allowing for end-to-end training via backpropagation without task-specific pre-training.
  • Improved QG for Better QA: The QG model, guided by the QA model's relevance judgments, learns to generate questions that are not just grammatically correct but also semantically aligned with the answer, which in turn helps the QA model.

Key Results

  • Improved QA and QG Performance: The proposed joint training framework consistently improves both QA and QG performance across all three datasets.
  • Competitive QA: The improved QA model achieves performance comparable to strong baseline approaches on MARCO and SQUAD. On WikiQA, the "Dual QA" model performs comparably to strong baseline methods.
  • QG Improvement: While BLEU-4 scores for QG are generally low (indicating the difficulty of the task), the joint training method shows a noticeable improvement over the basic QG model (e.g., from 8.87 to 9.31 on MARCO, and 4.34 to 5.03 on SQUAD).
  • Qualitative Examples: Sampled examples show that questions generated by the "Dual QG" model are more informative and overlap more with the ground truth questions compared to the "Basic QG" model, which helps the QA model.

Limitations and Future Work

  • Dataset Specificity: The authors acknowledge that MARCO and SQUAD, derived from reading comprehension tasks, might not be ideal for sentence-level QA and QG due to the answer being a span rather than a full sentence. They suggest that large-scale sentence-level QA datasets would be beneficial.
  • QG Model Limitations: The QG model doesn't explicitly use answer position information, which could be helpful for reading comprehension tasks. It also doesn't always capture the exact point of the reference question.
  • Evaluation of QG: The paper notes the difficulty of automatically evaluating QG systems beyond literal similarity (like BLEU-4) and suggests the need for metrics that assess whether a generated question is truly a question and answerable by the input.
  • Future Directions:
    • Refining individual QA and QG components with more complex architectures or external resources.
  • Developing neural language models for questions and answers that are jointly learned.
  • Investigating the use of duality during the inference process, not just training.
  • Exploring more sophisticated QG models that incorporate more answer information.

Summary updated

Abstract

We study the problem of joint question answering (QA) and question generation (QG) in this paper. Our intuition is that QA and QG have intrinsic connections and these two tasks could improve each other. On one side, the QA model judges whether the generated question of a QG model is relevant to the answer. On the other side, the QG model provides the probability of generating a question given the answer, which is a useful evidence that in turn facilitates QA. In this paper we regard QA and QG as dual tasks. We propose a training framework that trains the models of QA and QG simultaneously, and explicitly leverages their probabilistic correlation to guide the training process of both models. We implement a QG model based on sequence-to-sequence learning, and a QA model based on recurrent neural network. As all the components of the QA and QG models are differentiable, all the parameters involved in these two models could be conventionally learned with back propagation. We conduct experiments on three datasets. Empirical results show that our training framework improves both QA and QG tasks. The improved QA model performs comparably with strong baseline approaches on all three datasets.
BibTeX
@misc{tang2017question,
  author = {Duyu Tang and Nan Duan and Tao Qin and Ming Zhou},
  title = {{Question Answering and Question Generation as Dual Tasks}},
  year = {2017},
  eprint = {1706.02027},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.02027},
}

Loading related content...

Figure