Question Generation for Question Answering

Nan Duan, Duyu Tang, Peng Chen, Ming ZhouEMNLP 2017 · OpenReview

Abstract This research addresses the data scarcity challenge in Question Answering (QA) by leveraging large-scale, unlabeled Community-QA (CQA) data to automatically generate high-quality questions. By treating question generation (QG) as a dual task to answering, the authors develop a multi-stage engine that mines question patterns and predicts topics from source passages. Experimental results across SQuAD, MS MARCO, and WikiQA demonstrate that integrating these generated questions as an auxiliary signal significantly enhances the performance of state-of-the-art QA systems, particularly on datasets derived from real-world search logs.

The Data Scarcity Problem in QA

Modern Question Answering systems, whether based on knowledge bases or reading comprehension, rely heavily on large-scale, human-labeled QA pairs for training. However, manual labeling is prohibitively expensive and slow, leading to a scarcity of high-quality training data. The authors propose that question generation (QG) and question answering (QA) are two sides of the same coin: if a passage contains an answer, one should be able to "reverse-engineer" a relevant question from it. The challenge lies in generating questions that are grammatically correct, reflect common user intentions, and are grounded in the source text.

A Multi-Stage Question Generation Engine

The proposed system utilizes a multi-stage approach to transform passages into questions:

  1. Question Pattern Mining: Instead of generating text from scratch, the system mines "question patterns" (e.g., who founded # ?) from YahooAnswers. A clustering approach identifies common templates and their associated "historical topics"—the entities that typically fill the placeholders.
Examples of mined question patterns and associated historical topics used for generation.
  1. Question Pattern Prediction: The authors explored two neural approaches: a retrieval-based CNN that ranks existing patterns and a generation-based RNN (BiGRU) with an attention mechanism that generates patterns word-by-word.
  2. Question Topic Selection: The system identifies a phrase or entity from the passage to fill the pattern placeholder. In the RNN model, this is handled via a "COPY mechanism" based on attention scores.
  3. Question Ranking: A linear model selects the final question based on prediction scores, language model fluency, and word overlap.

Dual Learning for QA Enhancement

A key innovation is the use of "Dual Learning" intuition to improve existing QA systems. When evaluating an answer candidate $A$ for a question $Q$, the system generates a set of questions from $A$. If these generated questions are highly similar to the original question $Q$, the system's confidence in the answer candidate increases. This creates a powerful auxiliary signal for ranking potential answers.

Experimental Results

The generation-based model (G-QG) consistently outperformed the retrieval-based model (R-QG), achieving higher BLEU-4 scores across all datasets (e.g., 13.57 vs 11.38 on WikiQA). More importantly, integrating these generated questions led to consistent gains in QA performance across all benchmarks.

The most significant improvements were observed on the MS MARCO dataset, which consists of questions from real search logs. As shown in the table below, the integration of QG signals (QA+QG) improved the Mean Average Precision (MAP) from 0.5131 to 0.5230.

Impact of integrating generated questions (QA+QG) on MS MARCO performance metrics.

Conclusion and Future Work

This work demonstrates that Community-QA sites are a goldmine for training QG models that reflect real user intent. While the system shows strong results, the authors identified areas for improvement, particularly in handling passages with multiple facts (40% of errors) and improving topic selection accuracy. Future work will focus on more robust independent models for topic selection and deeper exploration of the dual relationship between QA and QG.

Summary updated

Abstract

This thesis proposes theoretical and numerical contributions to use Entropy-regularized Optimal Transport (EOT) for machine learning. We introduce Sinkhorn Divergences (SD), a class of discrepancies between probability measures based on EOT which interpolates between two other well-known discrepancies: Optimal Transport (OT) and Maximum Mean Discrepancies (MMD). We develop an efficient numerical method to use SD for density fitting tasks, showing that a suitable choice of regularization can improve performance over existing methods. We derive a sample complexity theorem for SD which proves that choosing a large enough regularization parameter allows to break the curse of dimensionality from OT, and recover asymptotic rates similar to MMD. We propose and analyze stochastic optimization solvers for EOT, which yield online methods that can cope with arbitrary measures and are well suited to large scale problems, contrarily to existing discrete batch solvers.
BibTeX
@inproceedings{duan2017question,
  author = {Nan Duan and Duyu Tang and Peng Chen and Ming Zhou},
  title = {{Question Generation for Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=HybTHGzdWH},
}
Figure