TL;DR
This paper introduces a novel framework for jointly training Question Answering (QA) and Question Generation (QG) models, treating them as "dual tasks." By leveraging the inherent probabilistic relationship between QA and QG, their proposed method improves the performance of both tasks simultaneously, achieving competitive results on standard benchmarks.
What Problem Does This Paper Address?
The paper tackles the challenge of improving Question Answering (QA) and Question Generation (QG) systems. While both tasks involve understanding the relationship between a question and an answer, they are typically trained independently. The authors propose that QA and QG are intrinsically linked – a good QA model can help a QG model generate more relevant questions, and a QG model can provide valuable signals for QA. The core problem is how to effectively exploit this "duality" to mutually enhance both tasks, rather than treating them as separate problems.
Methods
The key innovation is a joint training framework that explicitly models the probabilistic correlation between QA and QG. The authors formulate this relationship using Bayes' theorem, stating that the joint probability of a question-answer pair P(q, a) can be expressed as P(a)P(q|a) (related to QG) or P(q)P(a|q) (related to QA). Their framework aims to minimize a loss function that includes the individual losses for QA and QG, along with a regularization term that enforces this probabilistic duality.
Specifically:
- QA Model: A discriminative recurrent neural network (RNN) based model is used for answer sentence selection. It takes a question and a candidate answer sentence and outputs a score indicating their relevance.
- QG Model: A generative sequence-to-sequence (seq2seq) model is employed for question generation. It takes an answer sentence and generates a question.
- Joint Training: The models are trained simultaneously. The regularization term encourages the QA model to produce probabilities P(a|q) that are consistent with the QG model's P(q|a) and the language models P(a) and P(q). This is achieved by minimizing the squared difference between the log-probabilities derived from both sides of the probabilistic duality equation.
Datasets
The paper evaluates its approach on three datasets:
- MARCO: Originally for reading comprehension, it's adapted for answer sentence selection.
- SQUAD: Also originally for reading comprehension, adapted for answer sentence selection.
- WikiQA: A benchmark dataset specifically designed for answer sentence selection.
The authors note that MARCO and SQUAD are large-scale datasets, but their original task is different from answer sentence selection. They derive their answer sentence selection task by assuming that sentences containing the correct answer span are correct answers, and vice-versa.
Key Insights and Ideas
- QA and QG as Duals: The core insight is that QA and QG are not just related but can be viewed as dual tasks, offering a powerful avenue for mutual improvement.
- Probabilistic Duality as Regularization: Using the probabilistic correlation between QA and QG as a regularization term is an effective way to guide joint training, forcing the models to learn consistent representations.
- End-to-End Learning: The entire framework, including both QA and QG components, is differentiable, allowing for end-to-end training via backpropagation without task-specific pre-training.
- Improved QG for Better QA: The QG model, guided by the QA model's relevance judgments, learns to generate questions that are not just grammatically correct but also semantically aligned with the answer, which in turn helps the QA model.
Key Results
- Improved QA and QG Performance: The proposed joint training framework consistently improves both QA and QG performance across all three datasets.
- Competitive QA: The improved QA model achieves performance comparable to strong baseline approaches on MARCO and SQUAD. On WikiQA, the "Dual QA" model performs comparably to strong baseline methods.
- QG Improvement: While BLEU-4 scores for QG are generally low (indicating the difficulty of the task), the joint training method shows a noticeable improvement over the basic QG model (e.g., from 8.87 to 9.31 on MARCO, and 4.34 to 5.03 on SQUAD).
- Qualitative Examples: Sampled examples show that questions generated by the "Dual QG" model are more informative and overlap more with the ground truth questions compared to the "Basic QG" model, which helps the QA model.
Limitations and Future Work
- Dataset Specificity: The authors acknowledge that MARCO and SQUAD, derived from reading comprehension tasks, might not be ideal for sentence-level QA and QG due to the answer being a span rather than a full sentence. They suggest that large-scale sentence-level QA datasets would be beneficial.
- QG Model Limitations: The QG model doesn't explicitly use answer position information, which could be helpful for reading comprehension tasks. It also doesn't always capture the exact point of the reference question.
- Evaluation of QG: The paper notes the difficulty of automatically evaluating QG systems beyond literal similarity (like BLEU-4) and suggests the need for metrics that assess whether a generated question is truly a question and answerable by the input.
- Future Directions:
- Refining individual QA and QG components with more complex architectures or external resources.
- Developing neural language models for questions and answers that are jointly learned.
- Investigating the use of duality during the inference process, not just training.
- Exploring more sophisticated QG models that incorporate more answer information.
Summary updated