KnowYourNyms? A Game of Semantic Relationships

Ross Mechanic, Dean Fulgoni, Hannah Cutler, Sneha Rajana, Zheyuan Liu, Bradley Jackson, Anne Cocos, Chris Callison-Burch, Marianna ApidianakiEMNLP 2017

TL;DR

This paper introduces "KnowYourNyms?", a web-based game designed to crowdsource semantic relationships between words. By gamifying the process, the authors collect a large, high-quality dataset of word associations that can be used to train NLP models and provide insights into how humans understand word relationships.

What Problem Does This Paper Address?

Understanding the relationships between words (like synonyms, antonyms, hyponyms, and meronyms) is fundamental for many Natural Language Processing (NLP) tasks, such as text classification, sentiment analysis, and question answering. While resources like WordNet exist, they are not exhaustive and can be expensive and time-consuming to build and maintain. Traditional methods for acquiring this knowledge often rely on manual annotation or fully automatic classifiers, which can be slow, costly, or lack the nuance of human understanding. The authors aim to bridge this gap by leveraging the "games with a purpose" (GWAPs) methodology to efficiently and cost-effectively gather a large volume of accurately labeled lexical relationships.

Methods

The core of the paper is the "KnowYourNyms?" game, which employs a GWAP approach. Users are presented with a "base word" and a specific semantic relation type (e.g., "What are kinds of seafood?"). They have a short time limit (typically 20 seconds) to type as many related words as possible. The game dynamically generates input fields as users type. After the time is up, users see a scoring page that shows how their answers compare to those of other players, incentivizing them to provide answers that are common among the user base. This scoring mechanism is inspired by game shows like "Family Feud."

The system's backend is built using the Django framework, with a frontend developed using JavaScript, HTML, CSS, and libraries like d3.js for data visualization. The application is deployed on AWS.

For vocabulary selection, the authors carefully curate "base words" from WordNet, ensuring they have a sufficient number of related words in WordNet and are frequent enough in the Google n-grams corpus to avoid obscurity. They also implement a "skip" button for users who are stumped.

Datasets

The primary dataset generated by this work is the collection of semantic relationships provided by players of "KnowYourNyms?". This dataset is used to train and evaluate a semantic relation classifier.

The authors also reference existing datasets and resources:

  • WordNet: Used for selecting base words and for a "WordNet bonus" in scoring.
  • Google n-grams corpus: Used to filter for frequent words.
  • BLESS, EVALution, ROOT9, K&H+N: Datasets used for training relation detection systems prior to this work.
  • CONCRETE dataset: Used for correlating word difficulty with concreteness.

The paper does not introduce a new benchmark dataset in the traditional sense, but rather the data collected through the game serves as a valuable resource for training and evaluating NLP models.

Key Insights and Ideas

  • Gamification for Data Collection: The "games with a purpose" approach is highly effective for acquiring large amounts of high-quality semantic relationship data at no cost. The engaging nature of the game encourages sustained participation.
  • Human Perception of Word Relationships: The data collected provides broad insights into how people perceive relationships between words, offering valuable information for psychology and linguistics research.
  • Scoring as Social Proof: The scoring system, which rewards answers common among peers, encourages users to align their responses with the collective understanding, leading to more consistent and useful data.
  • Abstract vs. Concrete Words: The study found that abstract words are generally more difficult for users to provide semantic relations for than concrete words, a hypothesis supported by correlation analysis with the CONCRETE dataset.
  • The Value of Frequency: Filtering base words by frequency in large corpora is crucial to ensure user engagement and prevent frustration with obscure terms.

Key Results

  • The "KnowYourNyms?" game successfully collected over 15,000 user inputs from 160 crowdworkers, with many playing significantly more than the initial ten rounds.
  • A semantic relation classifier (LexNET) trained on this crowdsourced data achieved a weighted average F-Score of 0.34 on a test set. While this might seem modest, it's important to note the complexity of the task and the potential for improvement with more data.
  • Crowdworkers who played the game with pre-populated data (meaning their answers were compared against existing player data) rated the experience higher (4.46/5) and were more likely to play again (4.43/5) compared to those who played before any data was collected.
  • The correlation analysis confirmed that more abstract words are more difficult to handle and are skipped more often by users.

Limitations and Future Work

  • Vocabulary Difficulty: The authors acknowledge that some prompts, particularly those involving abstract words, are more difficult for users, leading to higher "pass" rates. They plan to use concreteness scores to filter out overly difficult words in the future.
  • Data Sparsity: Initially, users playing the game without prior data had lower scores, indicating the need for seeding the game with data or having a sufficient user base for meaningful scoring.
  • Future Work: The authors intend to analyze the collected relations in terms of typicality and gradual semantic category membership to enhance their utility for tasks like textual entailment. They also plan to continue expanding the dataset as more players engage with the game, which will further enrich understanding of human word perception and improve NLP models. The software and collected data are released under an open-source license for further research.

Summary updated

BibTeX
@inproceedings{mechanic2017knowyournyms,
  author = {Ross Mechanic and Dean Fulgoni and Hannah Cutler and Sneha Rajana and Zheyuan Liu and Bradley Jackson and Anne Cocos and Chris Callison-Burch and Marianna Apidianaki},
  title = {{KnowYourNyms? A Game of Semantic Relationships}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
}
Figure