Collective Intelligence in Citizen Science - A Study of Performers and Talkers

Ramine Tinati, Elena Simperl, Markus Luczak-Rösch, Max Van Kleek, Nigel ShadboltarXiv 2014 · OpenReview

TL;DR

This paper investigates the relationship between task completion and discussion participation in online citizen science projects, specifically on the Zooniverse platform. It reveals that a small, highly engaged subset of users drives a significant portion of both task completion and community discussion, highlighting the importance of fostering these "core communities" for scientific discovery.

What Problem Does This Paper Address?

Online citizen science projects leverage the collective intelligence of non-expert volunteers to tackle large-scale scientific problems, from classifying galaxies to analyzing biological data. While the "wisdom of the crowd" for task completion is well-studied, the role of emergent online communities and discussions within these projects is less understood. This paper addresses the gap in understanding how participation in these discussion forums influences task completion and whether these communities contribute to the overall scientific output. It aims to shed light on the dynamics of collective intelligence in these volunteer-driven scientific endeavors.

Methods

The authors employ a quantitative approach, analyzing a dataset of user activity from 10 Zooniverse projects spanning December 2009 to July 2013. Their analysis involves:

  • Content and Social Network Analysis: While not explicitly detailed in the provided text, the paper mentions using these methods to examine participation in online communities.
  • User Behavior Analysis: They analyze the relationship between the number of tasks completed and the number of discussion posts made by individual users.
  • Clustering: A k-means approach is used to identify distinct user types based on their frequency of discussion and task completion.
  • Role Identification: They analyze the content of forum discussions to identify emergent roles played by participants beyond simply asking and answering questions.

Datasets

The study utilizes a dataset from the Zooniverse platform, comprising data from December 2009 to July 2013. This dataset includes:

  • 250,000 users
  • Over 50 million tasks completed
  • 650,000 discussion posts

This is a newly compiled dataset for this specific cross-project analysis of Zooniverse.

Key Insights and Ideas

  • The "Core Community" Phenomenon: A surprisingly small fraction of users (around 40.5%) engage in both task completion and discussions. Even within this group, a highly active subset (2928 users) accounts for a disproportionately large amount of both task completion (29.0%) and discussion (72.0%). This suggests that a dedicated "core community" is crucial for driving scientific progress in these platforms.
  • Discussion Fuels Discovery: While the paper doesn't directly quantify the impact of discussion on new discoveries, it identifies emergent roles within forums like "Discoverer" and "Hypothesiser." These roles indicate that participants are not just asking for help but are actively engaging in scientific inquiry, proposing hypotheses, and conducting further investigation, suggesting a direct link between discussion and the scientific process.
  • Beyond Simple Task Completion: The analysis of roles in discussion forums reveals a richer picture than just answering questions. Participants act as informers, moderators, and even "cheerleaders," fostering a collaborative environment that likely supports sustained engagement and task completion.
  • Cross-Project Commonalities: The authors hint at a "noticeable cross-project phenomenon," suggesting that the dynamics of online citizen science communities and the evolution of language within them might be generalizable across different scientific domains.

Key Results

  • Task vs. Talk Ratio: Participants perform significantly more tasks than they post in discussions, with a median of 600 classifications versus 14 discussion posts per user.
  • Highly Active Users: A subset of users who completed at least 140 posts and 6000 classifications represented 29.0% of total classifications and 72.0% of total discussions.
  • Emergent Roles: Beyond basic Q&A, roles like "Discoverer" (asking questions leading to discoveries) and "Hypothesiser" (posing scientific questions or answers) were identified, indicating active scientific engagement within the community.

Limitations and Future Work

  • Awareness of Discussion Facilities: The study acknowledges that some participants may not be aware of the discussion forums, which could influence the observed participation rates.
  • Causality: While a strong correlation between discussion and task completion is observed, the paper doesn't definitively establish causality. It's possible that highly motivated individuals are simply active in both areas.
  • Future Work: The authors suggest further research into the "cross-project phenomenon," exploring how community language evolves with subject domains and how serendipitous discoveries are achieved through collective behavior. They also emphasize the broader implications for understanding collective intelligence research and the benefits of supporting and uniting online communities across different crowdsourcing systems.

Summary updated

Abstract

The recent emergence of online citizen science is illustrative of an efficient and effective means to harness the crowd in order to achieve a range of scientific discoveries. Fundamentally, citizen science projects draw upon crowds of non-expert volunteers to complete short Tasks, which can vary in domain and complexity. However, unlike most human-computational systems, participants in these systems, the `citizen scientists' are volunteers, whereby no incentives, financial or otherwise, are offered. Furthermore, encouraged by citizen science platforms such as Zooniverse, online communities have emerged, providing them with an environment to discuss, share ideas, and solve problems. In fact, it is the result of these forums that has enabled a number of scientific discoveries to be made. In this paper we explore the phenomenon of collective intelligence via the relationship between the activities of online citizen science communities and the discovery of scientific knowledge. We perform a cross-project analysis of ten Zooniverse citizen science projects and analyse the behaviour of users with regards to their Task completion activity and participation in discussion and discover collective behaviour amongst highly active users. Whilst our findings have implications for future citizen science design, we also consider the wider implications for understanding collective intelligence research in general.
BibTeX
@misc{tinati2014collective,
  author = {Ramine Tinati and Elena Simperl and Markus Luczak-Rösch and Max Van Kleek and Nigel Shadbolt},
  title = {{Collective Intelligence in Citizen Science - A Study of Performers and Talkers}},
  year = {2014},
  eprint = {1406.7551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1406.7551},
}
Figure