Comparing Neural Question Generation Architectures for Reading Comprehension

E. Margaret Perkoff, Abhidip Bhattacharyya, Jon Z. Cai, Jie CaoACL 2023

TL;DR

This paper compares three popular neural architectures (T5, BART, GPT-2) for generating questions from educational texts, finding T5 to be the most effective. It also investigates the impact of including question attributes (like "character" or "setting") in the input, which generally improves performance, especially for GPT-2.

What Problem Does This Paper Address?

Automated Question Generation (AQG) holds significant promise for reducing teacher workload and enhancing student learning by providing automated assessment tools and personalized tutoring. While AQG has been explored using generic text datasets, adapting these models to educational contexts (like textbooks and narrative stories) presents unique challenges due to variations in subject matter, student age, and discourse structure. Existing research often focuses on single model architectures or specific corpora. This paper aims to bridge this gap by systematically comparing the performance of three prominent neural architectures (T5, BART, GPT-2) on two distinct educational datasets (narratives and textbooks) and exploring how incorporating question attributes influences their effectiveness.

Methods

The researchers fine-tuned three pre-trained encoder-decoder or decoder-only language models: T5, BART, and GPT-2.

  • T5: Treated as a text-to-text model, where the encoder takes the task indicator, context, and answer as input, and the decoder generates the question.
  • BART: Used a conditional generation approach where the encoder receives the context, and the decoder uses the attribute and answer as a prefix to generate the question.
  • GPT-2: Employed a prompt-based approach, where the model receives a formatted string containing the context, answer, and optionally attributes, and generates the question.

A key aspect of their methodology was exploring the impact of including "question attributes" (e.g., "causal relation," "setting") as part of the input to these models, particularly for the narrative dataset. They also developed specific preprocessing steps for the Textbook Question Answering dataset to adapt it for text-based question generation.

Datasets

The study utilized two datasets:

  • FairytaleQA Corpus: A narrative dataset designed for K-12 reading comprehension, containing stories, questions, answers, and seven types of question attributes (character, setting, action, feeling, causal relationship, outcome resolution, prediction).
  • Textbook Question Answering (TQA) dataset: A dataset originally focused on middle school science textbooks, adapted for open-ended question generation. The authors implemented preprocessing steps to extract context, answers, and questions from this dataset.

Key Insights and Ideas

  • T5 Dominance: The T5 architecture consistently outperformed BART and GPT-2 across both narrative and textbook datasets, suggesting its strong adaptability to diverse educational text types.
  • Attribute Power: Including question attributes as input generally improved question generation quality, particularly for the GPT-2 model. This highlights the value of providing explicit semantic cues to guide the generation process.
  • Attribute Specificity: The benefit of attributes wasn't uniform. While "Setting" questions saw high scores across models, attributes like "Outcome Resolution" and "Character" showed more significant gains when attributes were included, especially for BART and GPT-2.
  • Domain Shift Matters: The significant performance drop on the TQA dataset compared to FairytaleQA underscores the challenges of adapting models to different domains and the need for domain-specific fine-tuning or more robust context representation.
  • Qualitative Nuances: Automated metrics like ROUGE and BLEU don't always capture subtle semantic errors or factual inaccuracies in generated questions, suggesting the need for human evaluation.

Key Results

  • Overall Performance: T5 achieved the highest RougeL scores, with 0.536 on FairytaleQA and 0.316 on TQA. BART outperformed GPT-2 on both datasets.
  • Attribute Impact: Incorporating attributes improved RougeL scores for most question types, with GPT-2 showing the most significant gains. For instance, the GPT model with attribute achieved a higher RougeL score than the baseline GPT model for most question attributes.
  • Setting Advantage: All models performed best on "Setting" attribute questions, achieving the highest RougeL scores.
  • TQA Challenges: Models performed significantly worse on the TQA dataset, with BART often producing shorter outputs or no output at all. GPT models sometimes hallucinated irrelevant words like "Hamburger."

Limitations and Future Work

  • Dataset Adaptation: The TQA dataset was adapted using automated methods, and human review could further refine its suitability for open-ended question generation.
  • Attribute Influence: While attributes helped, the models were primarily trained on inputs without them, limiting the full exploration of attribute-enhanced learning. Future work could explore more extensive attribute-rich data.
  • Context Representation: The TQA dataset's domain-specific nature and the need for nuanced context suggest that incorporating knowledge graphs or more complex discourse representations could be beneficial.
  • Evaluation: The authors acknowledge the limitations of automated metrics and plan to conduct human evaluations with educators and students to assess factual correctness, relevance, and fluency.
  • Prompting Strategies: Exploring alternative prompting strategies, such as fill-in-the-blank, for datasets like TQA is suggested.
  • Joint Generation: Investigating joint generation of question attributes and questions, rather than treating them separately, is another promising avenue.

Summary updated

BibTeX
@inproceedings{perkoff2023comparing,
  author = {E. Margaret Perkoff and Abhidip Bhattacharyya and Jon Z. Cai and Jie Cao},
  title = {{Comparing Neural Question Generation Architectures for Reading Comprehension}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure