TL;DR
This paper investigates how social networking platforms can be leveraged to solve the "participation bottleneck" in collective intelligence systems. By deploying Phrase Detectives—a game-with-a-purpose (GWAP) for linguistic annotation—on Facebook, the researchers demonstrate that a combination of social and financial incentives can produce high-quality data at a fraction of the cost of traditional crowdsourcing platforms like Amazon Mechanical Turk.
What Problem Does This Paper Address?
Collective Intelligence (CI) systems, such as Wikipedia or citizen science projects, rely on massive numbers of contributors to generate high-quality data. However, recruiting and retaining these volunteers is a significant hurdle. In the field of Natural Language Processing (NLP), creating annotated resources (like identifying what "it" refers to in a complex sentence) is a notorious bottleneck that usually requires expensive expert labor.
While Games-With-A-Purpose (GWAP) have successfully turned these tasks into entertainment, the authors argue that the next logical step is integrating these games into social networks. They seek to understand if the inherent social structures and data available on platforms like Facebook can provide better incentives for participation and whether these environments attract a specific demographic of "super-contributors."
Methods
The authors developed a Facebook-integrated version of Phrase Detectives, a game designed to collect data on anaphora resolution (identifying relationships between words and phrases).
Key technical and design features included:
- Two-Stage Workflow: The game uses "Annotation Mode" (Name the Culprit) for initial tagging and "Validation Mode" (Detective Conference) to resolve disagreements between players.
- Social Integration: Built using the Facebook PHP SDK, the game pulled friend lists to create "Teams," allowed automated "Wall" (News Feed) posting, and featured live leaderboards.
- Incentive Triad: The researchers tested three types of incentives:
- Personal: The intrinsic satisfaction of contributing to science and the enjoyment of reading interesting texts.
- Social: Earning points for agreeing with team members and gaining status via leaderboards.
- Financial: Monthly prizes (Amazon vouchers) for top scorers and lottery-style rewards.
- Quality Control: Players were initially vetted through "gold standard" training texts annotated by experts. Their ongoing "rating" (accuracy) influenced the points they earned and the weight of their contributions.
Datasets
The study analyzed logs from the Facebook implementation of Phrase Detectives over a one-year period (February 2011 to February 2012).
- Participants: 612 total players.
- Demographics: A subset of 112 active players who disclosed gender and completed at least one annotation was used for deeper behavioral analysis.
- Task Data: The game operated on a corpus of documents requiring linguistic annotation, comparing the Facebook-generated data against previous web-based versions of the game.
Key Insights and Ideas
- The Power of the Few: The workload follows a steep power-law distribution. A tiny elite of players (1.6%) produced a staggering 89% of the total annotations. This suggests that CI systems don't just need a "crowd"; they need to identify and retain "super-users."
- Gendered Engagement: Supporting previous social gaming surveys, the study found that women were more likely to play (63% of players) and, among the top tier of contributors, were significantly more active than men (p < 0.05).
- Social Sharing Preferences: When players shared their progress on their Facebook walls, they preferred "social" content (52% of posts were about the text they were reading) over purely competitive content like leaderboard positions (12%).
- Financial vs. Social Drivers: While financial prizes were excellent for recruitment and boosting the output of already active players, personal and social incentives provided a consistent baseline of activity even when prize funds were low.
Key Results
- Cost Efficiency: The average cost per annotation using financial incentives was £0.0028. This is significantly cheaper than Amazon Mechanical Turk, which typically costs between £0.007 and £0.70 per annotation.
- Recruitment: Active promotion and higher prize funds (e.g., July 2011) led to clear spikes in new player recruitment, doubling the numbers of quieter months.
- Annotation Modes: The game successfully funneled difficult tasks from Annotation Mode into Validation Mode, mimicking an expert review process through collective agreement.
Limitations and Future Work
- Goal Setting: The authors hypothesized that players would use level thresholds as specific goals (stopping once a level is reached). However, the data was inconclusive, partly because points are awarded in "batches" upon document completion, making it hard to stop exactly at a threshold.
- Quality Analysis: While female players produced more quantity, the authors noted that further research is required to determine if there are gender-based differences in annotation quality.
- Sustainability: Most players only participate for a short period. Future work should focus on long-term retention strategies to prevent the "handful of players" from burning out.
- Hidden Costs: The cost-per-annotation analysis did not fully account for the initial setup and ongoing technical maintenance of the CI platform itself.
Summary updated