Educational Multi-Question Generation for Reading Comprehension

Manav Rathod, Tony Tu, Katherine StasaskiNAACL 2022

TL;DR

This paper introduces "Multi-Question Generation" (MQG), a new task aimed at generating multiple semantically similar but lexically diverse questions from a single passage and answer. They propose an evaluation framework and demonstrate that while generating diverse questions is possible, it can sometimes come at the cost of answerability, highlighting a trade-off that needs careful consideration for educational applications.

What Problem Does This Paper Address?

Traditional automated question generation (QG) systems typically produce only one question per intended answer. While useful for reading comprehension, this single-question approach limits educational applications. Students benefit from encountering the same concept phrased in multiple ways to deepen understanding and practice without rote memorization. Furthermore, having varied question wordings is valuable for assessment, allowing for a question to be held out. This paper addresses the gap by proposing a method to generate multiple questions that assess the same concept but use different wording, aiming for semantic similarity and lexical diversity. A secondary problem they tackle is the tendency of existing QG models to generate questions with high lexical overlap with the source text, making it too easy for students to simply scan and find the answer.

Methods

The authors propose a new task, Multi-Question Generation (MQG), which takes a passage and an intended answer as input and outputs multiple semantically similar, lexically diverse questions. They develop an evaluation framework using three key metrics:

  • Question Answering (QA) Model Performance: They use a QA model to ensure the generated questions are answerable. This is measured by the F1 score, treating predicted and ground truth answers as bags of tokens.
  • PINC (Paraphrase In N-Gram Changes): This metric measures the lexical overlap between pairs of generated questions and between questions and the context passage. They use a symmetric version (PINCsym) for inter-question similarity and an asymmetric version to penalize questions that simply copy phrases from the context.
  • SBERT (Sentence-BERT) Cosine Similarity: This measures the semantic similarity between generated questions, ensuring they assess the same concept.

They explore several approaches for generating two questions (2QG):

  1. 1QG+Para: Generate one question with a standard QG model (ProphetNet) and then use a T5-based paraphrasing model to rephrase it.
  2. 2QG (Fine-tuning ProphetNet): Fine-tune ProphetNet to directly output two questions sequentially, separated by a special token.
  3. Constrained Generation: Introduce constraints to the 2QG model to encourage lexical diversity, such as preventing trigram repetition within questions (2QG No Question Trigram) or between questions and the context (2QG No Context Trigram).
  4. Sampling: Explore sampling from the model's learned distribution to generate multiple questions.

Datasets

The primary dataset used for training and evaluation is SQuAD 1.1. They augment this dataset for fine-tuning their 2QG model by creating paraphrased questions from a 1QG+Para approach.

Key Insights and Ideas

  • The Trade-off: A central finding is the inherent trade-off between generating lexically diverse questions (high PINC score) and ensuring those questions are answerable by a QA model (high QA score). This suggests that forcing too much lexical variation can sometimes lead to questions that are harder for machines (and potentially humans) to understand or answer correctly.
  • Answerability of Multiple Questions: The second question generated by some models (especially 1QG+Para) tends to have a lower QA score than the first. This implies that while the initial question might be well-formed, the subsequent rephrased or generated questions might suffer in terms of clarity or directness.
  • Contextual Constraints: Constraining questions to avoid trigrams from the context passage (2QG No Context Trigram) improves lexical diversity (higher PINC) with a smaller hit to QA performance compared to constraining questions to avoid repeating trigrams among themselves.
  • Semantic Similarity vs. Lexical Diversity: There's an inverse relationship between inter-question PINC score and SBERT similarity. This reinforces the trade-off: more lexical variation often means slightly less semantic overlap, which needs careful management.
  • Sampling for More Questions: While sampling from a 2QG model can generate more than two questions, the average between-question PINC score drops sharply after two questions, indicating that current sampling strategies might not effectively generate highly diverse sets beyond pairs.

Key Results

  • 2QG Performance: Table 3 shows that models like "2QG No Question Trigram" achieve high PINC scores (0.77) but at the cost of lower QA scores (0.63 for QA2). "2QG No Context Trigram" offers a better balance, with a good PINC score (0.75) and a relatively high QA score (0.77 for QA2).
  • Sampling Beyond Two: Figure 1 shows that the average between-question PINC score significantly decreases when sampling more than two questions from the 2QG model.
  • Qualitative Examples: Table 2 and Table 4 provide qualitative examples of generated question pairs, illustrating the concept of semantic similarity with lexical diversity and the potential for overlap with the context.

Limitations and Future Work

  • Human Evaluation: The authors acknowledge that their current evaluation metrics are heuristic and limited. They suggest future work should include human evaluation to more robustly assess semantic quality and lexical diversity.
  • Reinforcement Learning: They propose using desirable question metrics within a reinforcement learning objective to guide the generation of higher-quality questions.
  • Advanced Paraphrasing: Leveraging more sophisticated paraphrase systems, like syntax-aware ones, could lead to better lexical variation.
  • Educational Impact Study: A significant future direction is to integrate MQG into educational environments to study the actual impact of multiple question wordings on student comprehension and learning.
  • Beyond Two Questions: Developing better methods for generating sets of more than two diverse questions is a key area for future research.

Summary updated

BibTeX
@inproceedings{rathod2022educational,
  author = {Manav Rathod and Tony Tu and Katherine Stasaski},
  title = {{Educational Multi-Question Generation for Reading Comprehension}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure