Tabouid: a Wikipedia-based word guessing game

Timothée BernardACL 2020

TL;DR

This paper introduces "Tabouid," a word-guessing game that automatically generates its content from Wikipedia articles. By leveraging NLP and machine learning, it creates a playable game with a difficulty score for each card, making it adaptable for various players and potentially useful for language learning.

What Problem Does This Paper Address?

While Wikipedia is a vast repository of information used for many AI and NLP tasks (like knowledge bases and question-answering datasets), its potential for creating engaging, educational games is largely untapped. The authors aim to bridge this gap by developing a system that can automatically transform Wikipedia articles into playable content for a word-guessing game, inspired by the popular game "Taboo." This addresses the challenge of creating a large, diverse, and dynamically generated game content without manual curation.

Methods

The core of Tabouid's content generation lies in a two-stage process:

  1. Page Scoring (Difficulty Estimation):
    • A neural network is trained to predict a "difficulty" or "playability" score for Wikipedia pages.
  2. Page Representation: Each page is represented by a vector derived from:
  3. Categories and Title Features: Information about the page's categories and its title.
  4. Various Features: 10 features including page visit counts (and their variance), page size, creation/modification dates, number of translations, and whether the title contains month names, dates, or foreign characters.
  5. Dimensionality Reduction: Truncated SVD is used to reduce the dimensionality of category and title features.
  6. Neural Network Architecture: A simple neural network with interpretable parameters is used to prevent overfitting, with an initialisation strategy based on intuitions about feature impact.
  7. Training: The network is trained using an active learning approach on a manually annotated subset of Wikipedia pages, aiming for a balanced dataset.
  8. Banned Words Selection:
    • For pages deemed suitable (easier ones), a list of banned words is extracted.
  9. Process: * Wikipedia article text is tokenized, stemmed, and stop words are removed.
  10. The 10 most frequent stems are identified.
  11. The most frequent token for each of these stems is selected as a banned word.
  12. Enhancements: * Custom rules are added to the stemmer to group related words (e.g., "lawyer" and "law").
  13. Longer expressions (titles of linked Wikipedia articles) are also considered banned.
  14. Linked entities are weighted higher (1.5x) to account for their importance.
  15. The final selection prioritizes entity names over individual words if both are candidates.
  16. Output: Each card consists of a title (the Wikipedia page title) and a list of 10 banned words or expressions.

Datasets

  • Wikipedia: The primary source for generating game content. The paper mentions using a large subset of Wikipedia articles.
  • Manually Annotated Data: A dataset of approximately 2300 Wikipedia pages were manually scored for difficulty/playability. This dataset was used to train the neural network. The authors employed an active learning strategy to efficiently collect these annotations.

Key Insights and Ideas

  • Wikipedia as a Game Content Engine: The paper demonstrates that Wikipedia's vast and diverse content can be automatically transformed into engaging game material, overcoming the bottleneck of manual content creation.
  • Difficulty Scoring is Crucial: Not all Wikipedia pages make good game cards. Pages that are too obscure or too technical are unsuitable. The paper's focus on scoring pages for "playability" is key to filtering for good content.
  • Active Learning for Annotation: Manually annotating data for game playability is inefficient. The active learning approach, starting with high-traffic pages and iteratively selecting uncertain instances, is an effective strategy for building a useful training set.
  • Interpretable Neural Networks: Using a simple neural network with interpretable parameters and a sensible initialisation strategy can lead to better performance and more reliable training, especially with limited annotated data.
  • Pedagogical Potential: Beyond entertainment, Tabouid has potential as an educational tool for language learners and for teaching NLP concepts through implementation projects.

Key Results

  • Game Content: Tabouid contains 10,000 cards in English and French, covering a wide range of topics.
  • Neural Network Performance:
    • The best accuracy achieved by the trained neural network for binary classification (easy/hard) was 85.0% with interpretable weights initialised.
  • The mean accuracy was 83.5% with a standard deviation of 1.1%.
  • Models without the initialisation procedure performed slightly worse (best accuracy 84.4%, mean 82.9%).
  • Feature Importance: While the paper doesn't detail all feature weights, it notes that some intuitions about feature impact (like page popularity) were validated, while others (like page size and number of translations) had surprisingly low direct impact on the score.
  • Banned Word Selection: The algorithm successfully generates lists of banned words, though it acknowledges potential for missing important words or selecting distinct forms of the same word. The use of 10 banned words per card is a strategy to mitigate limitations.

Limitations and Future Work

  • Banned Word Selection Imperfections: The current banned word selection algorithm is not perfect. It may miss important words or incorrectly select distinct forms of the same word. The authors compensate by providing 10 banned words per card.
  • Content Appropriateness: Some cards may contain content deemed inappropriate for children.
  • Future Work:
    • Improving the banned words selection process.
  • Generating specific card lists for educational programs.
  • Allowing users to select cards based on Wikipedia categories.
  • Leveraging Wikipedia's multilingual nature to generate content in more languages.
  • Predicting and filtering cards based on age appropriateness.

Summary updated

BibTeX
@inproceedings{bernard2020tabouid,
  author = {Timothée Bernard},
  title = {{Tabouid: a Wikipedia-based word guessing game}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure