Enhancing Human Summaries for Question-Answer Generation in Education

Hannah Gonzalez, Liam Dugan, Eleni Miltsakaki, Zhiqi Cui, Jiaxuan Ren, Bryan Li, Shriyash Upadhyay, Etan Ginsberg, Chris Callison-BurchACL 2023

TL;DR

This paper tackles the challenge of generating high-quality educational questions from text. They found that while large language models (LLMs) like GPT-3 are improving, human-written summaries still yield better questions than LLM-generated ones. Their novel method, Bull2Sum, effectively converts student bullet-point notes into richer summaries, leading to significantly improved question generation quality.

What Problem Does This Paper Address?

Automated question generation (QG) holds great promise for education, offering tools for self-review and assessment creation. However, a critical hurdle is ensuring the relevancy of generated questions, not just their grammatical correctness. Previous research by Dugan et al. (2022) demonstrated that using summaries as input to QG models significantly boosts question relevance compared to using raw textbook text. Crucially, human-written summaries outperformed automatically generated ones. This paper investigates whether recent advancements in LLMs have closed this gap and introduces a new method to improve the quality of automatically generated summaries for QG.

Methods

The core of this work lies in two main areas: evaluating LLM summarization capabilities and introducing a novel summarization technique.

  • LLM Summarization Evaluation: The authors first assess whether state-of-the-art LLMs, specifically GPT-3, can now generate summaries that rival human quality for QG. They compare GPT-3's zero-shot, few-shot, and fine-tuned summarization outputs against human-written summaries.
  • Bull2Sum Methodology: The paper introduces "Bull2Sum," a novel methodology designed to transform student-generated bullet-point notes into fully-fledged, coherent summaries. This is achieved by fine-tuning an LLM (GPT-3) on a dataset of bullet-point notes and their corresponding expanded summaries. The process involves a bootstrapping approach:
  • Zero-shot: Initial summaries are generated using a prompt.
  • Few-shot: Manually reviewed and filtered examples are used to create a prompt for generating more summaries.
  • Fine-tuning: The model is fine-tuned on the few-shot generated data.

These generated summaries are then used as input for question generation models.

Datasets

The researchers collected and utilized several datasets:

  • Human-written Summaries: Summaries from 570 undergraduate and Master's students enrolled in an Artificial Intelligence course. These were categorized into "fully-fledged summaries" (coherent paragraphs) and "bullet-point summaries" (notes). A subset of 392 students' anonymized summaries was released.
  • Student Bullet-Point Notes: These served as the input for the Bull2Sum model.
  • Newly Introduced Benchmark: The paper releases a novel dataset of 9.2K human annotations of generated question-answer pairs. This dataset is intended to aid future research in evaluating QG models.

Key Insights and Ideas

  • LLMs Still Lag Behind Humans for QG Summaries: Despite advancements, GPT-3-generated summaries, even when fine-tuned, did not consistently outperform human-written summaries in producing high-quality QG. This suggests that the nuances required for educational summarization for QG are still best captured by human authors.
  • Student Notes are a Rich, Untapped Resource: The paper highlights that student bullet-point notes, often considered informal, can be a valuable source of information. The challenge is transforming them into a format suitable for QG.
  • Bull2Sum Bridges the Gap: The Bull2Sum methodology proves effective in converting these informal notes into richer summaries, significantly boosting the quality of generated questions. This offers a practical way to leverage existing student study materials.
  • Quality Over Quantity: The results emphasize that the quality of the input summary is more critical than its length or complexity for generating relevant and useful questions.

Key Results

The study involved a large-scale human evaluation of generated question-answer pairs. Key findings include:

  • GPT-3 vs. T5: The larger LLM, GPT-3, generally outperformed the T5 model in question generation across various input types, producing more acceptable, grammatical, interpretable, relevant, and correct QA pairs.
  • Summary Input Performance:
    • Fully-fledged human-written summaries yielded the best QA pairs (73% acceptability).
  • Bull2Sum summaries achieved a substantial 70% acceptability, coming very close to human-written ones. This is a significant improvement over raw bullet-point summaries (53% acceptability).
  • GPT-3 generated summaries (from textbook text) performed moderately well but were still surpassed by human summaries and Bull2Sum.
  • Bull2Sum's Impact: The Bull2Sum method resulted in a substantial increase in QA pair quality, particularly in acceptability, improving from 53% with bullet points to 70% with Bull2Sum summaries.

Limitations and Future Work

  • Domain Specificity: The research focused on computer science textbooks. The effectiveness of the proposed methods on other disciplines, which might have different knowledge structures or internet prevalence, remains to be explored.
  • Question Type Diversity: The study primarily examined answer-agnostic question generation. Exploring other question types (e.g., multiple-choice, compare-and-contrast) would require further adaptation of the methodology.
  • Factually Incorrect Information: As with all LLMs, there's a risk of generating factually incorrect information. The authors note that even their best-performing models achieved only 70% acceptability, underscoring the need for manual review before deployment in educational settings.
  • Intended Use: The models and methods are presented for research purposes and should not be deployed as standalone solutions without further evaluation.
  • Future Directions: The authors suggest that Bull2Sum could be utilized to convert course slides into summaries, thereby generating relevant questions to aid student exam preparation.

Summary updated

BibTeX
@inproceedings{gonzalez2023enhancing,
  author = {Hannah Gonzalez and Liam Dugan and Eleni Miltsakaki and Zhiqi Cui and Jiaxuan Ren and Bryan Li and Shriyash Upadhyay and Etan Ginsberg and Chris Callison-Burch},
  title = {{Enhancing Human Summaries for Question-Answer Generation in Education}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure