QGASP: a Framework for Question Generation Based on Different Levels of Linguistic Information

Hugo Rodrigues, Luísa Coheur, Eric NybergINLG 2016 · DOI

TL;DR

This paper introduces QGASP, a novel framework for question generation that leverages lexical, syntactic, and semantic information to learn patterns and generate questions. By integrating these multiple linguistic levels, QGASP aims to improve the quality and variability of generated questions compared to systems relying on simpler pattern types.

What Problem Does This Paper Address?

Generating high-quality questions automatically from text is a challenging task. Existing systems often rely on lexico-syntactic patterns, which can limit the expressiveness and semantic richness of the generated questions. The authors identify a gap in leveraging a broader spectrum of linguistic information, specifically semantic features, to enhance both the learning of question generation patterns and the generation process itself. This work aims to bridge this gap by proposing a framework that integrates lexical, syntactic, and semantic information at multiple stages of question generation.

Methods

QGASP's architecture consists of two main stages: Pattern Acquisition (PA) and Question Generation (QG).

  • Pattern Acquisition: This stage learns patterns from "seeds," which are triples containing a question, its answer (optional), and a snippet that could answer the question. Both the question and snippet are processed using Stanford's syntactic and dependency parsers, and MatePlus's Semantic Role Labeler (SRL). A "pattern" is defined as a bidirectional mapping between subtrees of the question and the snippet.
  • Question Generation: Given a new sentence, QGASP first parses it. Then, it matches the previously learned patterns against the parsed sentence's structures.
  • Matching Step: A key innovation is the matching strategy, which operates at lexical, syntactic, and semantic levels. This allows for a flexible matching approach:
  • Lexical Match: Checks if token lemmas match (using WordNet).
  • Semantic Match: Compares verb SRLs and uses a verb dictionary mapping PropBank, VerbNet, and FrameNet. Non-verb tokens are considered a semantic match if they belong to the same WordNet synset. Named Entities (NEs) match if they have the same type according to Stanford NER.
  • Syntactic Match: Compares subtree structures for identical labels and number of children.

This multi-level matching aims to strike a balance between capturing too much noise (loose matching) and being too restrictive (specific patterns leading to low variability).

Datasets

  • Engarte corpus: Used for testing QGASP. The corpus provided 32 revised triples labeled as "true." These triples were used for both Pattern Acquisition and Question Generation.
  • WordNet: Used for lexical matching and identifying semantic similarity between non-verb tokens.
  • PropBank, VerbNet, FrameNet (via SemLink): Used for semantic matching of verbs.
  • Stanford Named Entity Recognition (NER): Used for semantic matching of Named Entities.

The evaluation used a "leave one out" approach, meaning a pattern learned from a specific sentence was not applied to that same sentence during the QG phase.

Key Insights and Ideas

  • Semantic Information is Crucial: The paper strongly advocates for the integration of semantic information, arguing it's essential for learning richer patterns and generating more nuanced questions.
  • Multi-Level Matching for Flexibility: The flexible matching strategy (lexical, syntactic, semantic) is a core contribution, allowing QGASP to adapt to different levels of similarity and avoid the pitfalls of overly strict or loose matching.
  • Bidirectional Patterns: Defining patterns as bidirectional mappings between question and snippet subtrees allows for a more robust representation of the relationship between text and potential questions.

Key Results

  • Pattern Acquisition: 23 semantic patterns were learned from the Engarte corpus.
  • Question Generation Evaluation: 201 questions were generated.
  • 92% were considered plausible (of any sort).
  • 184 were considered plausible (excluding those needing context).
  • 32 were labeled as plausible needing context.
  • Agreement: Cohen's Kappa agreement on a subset of 115 questions was 0.67, indicating substantial agreement among annotators.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that "several points can be improved in QGASP."
    • The evaluation was conducted on a relatively small dataset (Engarte corpus, 32 triples).
  • The "plausible needing context" category suggests that some generated questions might require additional context to be fully understood, indicating room for improvement in question clarity.
  • Future Work: Not explicitly detailed in the provided text, but the authors suggest that demonstrating how seeds are learned and how semantic features improve QG is a step towards further development. The multi-level approach itself opens avenues for exploring different weighting or combinations of these levels.

Summary updated

BibTeX
@inproceedings{rodrigues2016qgasp,
  author = {Hugo Rodrigues and Luísa Coheur and Eric Nyberg},
  title = {{QGASP: a Framework for Question Generation Based on Different Levels of Linguistic Information}},
  booktitle = {INLG},
  year = {2016},
  doi = {10.18653/v1/w16-6640},
}
Figure