Automatic Question Generation for Vocabulary Assessment

Jonathan Brown, Gwen Frishkoff, Maxine EskenaziEMNLP 2005

TL;DR

This paper introduces a system for automatically generating vocabulary assessment questions using WordNet. The generated questions, covering six types like definition, synonym, and cloze, correlate well with human-written assessments and standardized tests, offering a more efficient and objective way to measure vocabulary knowledge.

What Problem Does This Paper Address?

Assessing vocabulary knowledge is crucial for personalized learning systems like REAP, which aims to provide readers with texts tailored to their reading level and vocabulary. Traditionally, vocabulary assessments have been handcrafted, a process that is time-consuming and often subjective. The authors tackle the challenge of creating an automated, objective, and fine-grained method for vocabulary assessment that can capture different levels of word mastery, moving beyond simple all-or-none knowledge.

Methods

The core of the paper is the REAP system's automatic question generation engine. This system leverages WordNet, a lexical database, to extract information about words. For a given target word, the system can generate six types of questions:

  • Definition: Uses WordNet glosses.
  • Synonym: Matches the target word with a synonym from the same synset or related synsets.
  • Antonym: Matches the target word with an antonym, either direct or indirect.
  • Hypernym: Identifies the generic term for a class (e.g., "organism" for "person").
  • Hyponym: Identifies members of a class (e.g., "adult" for "person").
  • Cloze: Uses example sentences or phrases from WordNet with the target word blanked out.

These questions can be presented in two forms: wordbank (matching items to a list) and multiple-choice. The system also employs strategies for selecting distractors (incorrect answer choices) based on part-of-speech and word frequency, drawing from the British National Corpus (BNC).

Datasets

  • WordNet: Used as the primary lexical resource for generating question content.
  • British National Corpus (BNC): Used via the Kilgarriff's word frequency database for selecting distractors.
  • 156 low-frequency and rare English words: Used for initial question coverage analysis.
  • 75 low-frequency English words: Used for experimental evaluation of the generated questions.
  • Standardized measures: Nelson-Denny Reading Test, Raven's Matrices Test, and Lexical Knowledge Battery were used as external benchmarks for validation.

Key Insights and Ideas

  • WordNet as a rich source for question generation: The paper demonstrates that WordNet's structured lexical information (definitions, synonyms, antonyms, hypernyms, hyponyms) is sufficient to automatically create a variety of vocabulary assessment questions.
  • Graded vocabulary knowledge: The authors acknowledge that word knowledge isn't binary and aim to assess different levels of mastery, aligning with models like Stahl's. The different question types are hypothesized to tap into these different levels.
  • Importance of good distractors: The paper highlights the careful consideration given to selecting distractors that are semantically and statistically similar to the correct answer, a crucial aspect for effective multiple-choice questions.
  • Automated assessment can rival human assessment: The strong correlations found between computer-generated and human-generated questions suggest that automated systems can provide a valid measure of vocabulary skill.

Key Results

  • Question Coverage: The system could generate questions for 91% of words for definition, 80% for synonym, and 60% for antonym and cloze types.
  • Validity: Computer-generated questions showed strong correlations with human-generated questions across various types (e.g., synonym: r=.906, cloze: r=.860).
  • Standardization: Accuracy on computer-generated questions correlated well with scores on standardized vocabulary tests (Nelson-Denny), indicating they tap into general vocabulary knowledge.
  • Specificity: Correlations with the Raven's Matrices test (non-verbal reasoning) were low and not statistically significant, suggesting the questions specifically measure vocabulary, not general intelligence.
  • Difficulty Ordering: The computer-generated definition task had the highest accuracy (.6452), while the cloze task was the hardest (.5286), aligning with expected difficulty levels.

Limitations and Future Work

  • Limited Question Types: The study focused on four of the six generated question types (definition, synonym, antonym, cloze) due to coverage issues with hypernym/hyponym for adjectives.
  • Generation of Higher-Level Questions: The current system does not assess "generation processing" (e.g., writing a sentence using a word in a novel context), which requires free-form responses.
  • Distractor Strategies: While good distractors were used, further exploration of more complex strategies is suggested.
  • Word Sense Disambiguation: The system currently relies on frequency or manual annotation for word sense. Future work will incorporate word sense disambiguation techniques.
  • Assessing Different Word Lists: The current study used low-frequency words; future work will assess questions on word lists with different characteristics.
  • Confidence Ratings: The low correlation between confidence ratings and accuracy suggests further investigation is needed to determine their predictive power for vocabulary knowledge.

Summary updated

BibTeX
@inproceedings{brown2005automatic,
  author = {Jonathan Brown and Gwen Frishkoff and Maxine Eskenazi},
  title = {{Automatic Question Generation for Vocabulary Assessment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure