Rating Computer-Generated Questions with Mechanical Turk

Michael Heilman, Noah A. SmithNAACL 2010 · OpenReview

Abstract

We use Amazon Mechanical Turk to rate computer-generated reading comprehension questions about Wikipedia articles. Such application-specific ratings can be used to train statistical rankers to improve systems' final output, or to evaluate technologies that generate natural language. We discuss the question rating scheme we developed, assess the quality of the ratings that we gathered through Amazon Mechanical Turk, and show evidence that these ratings can be used to improve question generation.
BibTeX
@inproceedings{heilman2010rating,
  author = {Michael Heilman and Noah A. Smith},
  title = {{Rating Computer-Generated Questions with Mechanical Turk}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=SkbplX-ubS},
}

Loading related content...

Figure