Phrase Detectives: Utilizing Collective Intelligence for Internet-Scale Language Resource Creation (Extended Abstract)

Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, Luca DucceschiIJCAI 2015

Overview

Phrase Detectives is a "game-with-a-purpose" (GWAP) designed to solve the massive data bottleneck in Natural Language Processing (NLP) by turning linguistic annotation into an engaging detective mystery. By leveraging collective intelligence, the project created a high-quality corpus of anaphoric coreference at a fraction of the cost of expert annotators, achieving 84% agreement with professionals.

The Annotation Bottleneck

Modern Human Language Technology (HLT) relies heavily on supervised machine learning, which requires enormous amounts of annotated data. One of the most challenging tasks to annotate is anaphora coreference—identifying when different words refer to the same entity (e.g., recognizing that "Jon," "him," and "the cyclist" all refer to the same person in a story).

Traditionally, this process is conducted by experts, which is prohibitively expensive, often estimated at $1.00 per token. While crowdsourcing platforms like Amazon Mechanical Turk (MTurk) offer a cheaper alternative, they still incur significant costs and require complex management to ensure data quality. The authors address the need for a sustainable, "internet-scale" method to generate these linguistic resources for free or very low cost.

The Phrase Detectives Game

The authors developed Phrase Detectives, an interactive web-based game where players take on the role of a detective to solve "cases" (linguistic ambiguities). The game's interface is designed to be engaging, featuring leaderboards, user profiles, and progress tracking to maintain player interest.

The Phrase Detectives game interface, featuring leaderboards and user profiles to encourage engagement.

The system uses two primary mechanics to ensure data collection and accuracy:

  1. Annotation Mode ("Name-the-Culprit"): Players identify the antecedent of a highlighted markable. If multiple players agree on an interpretation, the task is considered complete.
  2. Validation Mode ("Detectives Conference"): If players disagree in the first mode, the case moves to validation. Other players review the proposed interpretations and vote on them.

To filter out "sloppy or malicious" players, the authors implemented an Annotation-Validation (AV) model. They use a simple but effective formula to score interpretations: $Ann + Agr - Disagr$ (Initial annotations + Agreements - Disagreements). Spurious or "cheated" entries typically result in a zero or negative score, allowing the system to extract high-quality data from non-experts.

Datasets and Scale

The project utilized several sources to create its corpus, ensuring a mix of narrative and factual styles:

  • The Gutenberg Project: Narrative texts used for training and initial evaluation.
  • Wikipedia: Encyclopedic texts used to test the system's performance on more factual, structured data.
  • Comparison Benchmarks: Results were evaluated against established corpora like MUC6, ACE2, and OntoNotes to see how the "game-generated" data performed when training anaphora resolution algorithms (such as the BART toolkit).

At the time of the study, the English version contained over 1.2 million words across 839 documents.

Key Results and Cost-Effectiveness

The Phrase Detectives project demonstrated that gamification is a viable path for large-scale linguistic resource creation. Key findings include:

  • Expert Agreement: The "game interpretation" (the majority vote of players) matched expert judgments in 84% of cases.
  • Scale of Contribution: Over 8,000 players registered, contributing over 5,000 hours of work—the equivalent of 2.5 person-years of effort.
  • Model Performance: Anaphora resolution models trained on the game's data achieved an F-value of 0.58 on Gutenberg texts and 0.49 on Wikipedia texts. While slightly lower than models trained on expert data, the sheer volume of data the game can produce helps bridge the gap.
  • Throughput: The game averaged 450 annotations per hour.

One of the most striking results is the cost-effectiveness of the GWAP approach. As shown in the table below, the cost per markable for Phrase Detectives is significantly lower than traditional or even other crowdsourced methods.

Comparison of costs per markable across different annotation methods.

High-quality expert annotation costs ~$3.00 per markable. Phrase Detectives achieved comparable results for **$0.47 per markable**, primarily accounting for server maintenance and small prizes.

Future Directions

The primary limitation identified is speed. At the current rate of 34,000 completed markables per year, it would take nearly a decade to finish the current backlog of 300,000+ markables. To accelerate this, the authors are focusing on:

  • Facebook Integration: Leveraging social features to attract a larger, more consistent player base through competition and social pressure.
  • Advanced Post-processing: Using physical indicators (like time spent on a task) and more complex aggregation techniques to further filter out "outlier" players.
  • Data Release: Making the completed "gold standard" sub-corpus available to the wider research community via the Linguistic Data Consortium (LDC).

Summary updated

BibTeX
@inproceedings{poesio2015phrase,
  author = {Massimo Poesio and Jon Chamberlain and Udo Kruschwitz and Livio Robaldo and Luca Ducceschi},
  title = {{Phrase Detectives: Utilizing Collective Intelligence for Internet-Scale Language Resource Creation (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure