Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts

Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, Teruko MitamuraCOLING 2016 · OpenReview

Abstract

Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, Teruko Mitamura. Proceedings of COLING 2016.

Researchers at Carnegie Mellon University have developed a system that generates multiple-choice questions requiring "deep" reading comprehension by linking information across multiple sentences. By leveraging semantic analysis—specifically coreference resolution and paraphrase detection—the system moves beyond simple surface-level pattern matching to create questions that truly test a learner's ability to connect the dots.

Introduction: The "Shallow" Question Problem

Most automated question generation (QG) systems are "shallow." They typically take a single sentence and apply syntactic transformations to turn it into a question (e.g., "The cat sat on the mat" becomes "Where did the cat sit?"). While these systems produce grammatically correct questions, they don't actually measure reading comprehension. A student can often answer them just by finding the sentence with the same keywords.

This paper addresses the need for higher-level questions—those that require "inference steps" across a text. To answer these, a reader must understand that a pronoun in sentence two refers to a complex process mentioned in sentence one, or recognize that two different phrases are actually describing the same event.

System Architecture

The proposed system follows a pipeline that starts with text and ends with multiple-choice questions, incorporating target selection, question construction, and distractor generation.

Overview of the automated question generation pipeline.

Methodology

The authors propose a two-pronged approach to question generation, complemented by a novel distractor generation strategy.

QG1: Leveraging Semantic Glue

This system uses templates to find "semantic glue" between sentences. It looks for event coreference (e.g., "divide" in one sentence and "division" in another) and entity coreference (e.g., "a few tumor cells" and "they"). By identifying these links, the system can ask a question using information from one sentence where the answer resides in another.

Semantic relations and corresponding question patterns and templates used in QG1.

QG2: Concept Relationships

This is a more flexible, pattern-based system. It extracts generic patterns from existing human-generated Q&A pairs (like "What happens because of [Entity] [Trigger]?") and attempts to apply them to new passages based on the relationships between events and entities.

Distractor Generation

To create a multiple-choice question, plausible wrong answers are required. The system builds an event graph of the entire passage. It identifies the "nodes" (events) used in the question and the correct answer, then selects other events from the graph to turn into phrase-level distractors. This ensures the distractors are contextually relevant but factually incorrect for that specific question.

Evaluation

The study utilizes the ProcessBank corpus, which consists of 200 paragraphs about biological processes extracted from a high school biology textbook. Biology is particularly suitable as it is heavy on causal chains and complex sequences. ProcessBank includes high-quality human annotations for event triggers, entity mentions, coreference links, and causal relations, allowing the researchers to focus on the logic of question generation.

The researchers compared their systems against a well-known baseline (Heilman and Smith, 2010):

  • Inference Steps: QG1 achieved an average of 0.76 inference steps per question, significantly higher than the baseline's 0.16.
  • Grammaticality: Despite the increased complexity, QG1 maintained a grammatical correctness score (1.50 on a 1-3 scale, where 1 is best) comparable to the baseline (1.34).
  • Distractor Quality: Human evaluators rated the distractors around 1.94 on a 3-point scale, indicating they were generally viable but incorrect.

Conclusion

The authors demonstrate that coreference resolution is the key to breaking the "single-sentence barrier" in automated QG. By linking mentions across sentences, the system forces the learner to maintain a mental model of the text. While currently reliant on expert annotations, future work aims to integrate automated semantic parsers to create a fully end-to-end system capable of generating deep questions from raw text.

Summary updated

BibTeX
@inproceedings{araki2016generating,
  author = {Jun Araki and Dheeraj Rajagopal and Sreecharan Sankaranarayanan and Susan Holm and Yukari Yamakawa and Teruko Mitamura},
  title = {{Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=1mNL27fVCXZ},
}
Figure