Crowd Memory: Learning in the Collective

Walter S. Lasecki, Samuel White, Kyle I. Murray, Jeffrey P. BighamarXiv 2012 · arXiv

Abstract

Traditional crowdsourcing models often treat workers as "stateless" entities, assuming they perform independent micro-tasks without prior context. This paper challenges that assumption by investigating "crowd memory"—the capacity for a collective to retain individual knowledge and pass it to new members. Through experiments involving grid-based search and real-time maze navigation, the authors demonstrate that crowds can function as a single, knowledgeable agent capable of maintaining high performance despite high worker turnover.

Introduction

Most crowdsourcing algorithms are designed for simple, independent tasks where workers arrive, complete a task, and leave without needing context. However, this limits the crowd's potential for complex, continuous tasks like technical support or strategic navigation. This research explores whether crowds can maintain "memory" at two levels: individual retention (returning workers remembering past solutions) and collective learning (the group maintaining knowledge as members are replaced).

Methods

The researchers conducted two sets of experiments to test these hypotheses:

  1. Individual Memory Retention: Using Amazon Mechanical Turk, workers were tasked with finding "gold" behind one of nine doors in a 3x3 grid. Performance was measured by the number of clicks required. Workers were re-recruited at intervals up to 12 hours to measure how much of the learned solution they retained.
  2. Collective Learning in Legion: Using the Legion system, a crowd controlled a single character in a Half-Life 2 maze. The task required navigating to and pressing specific buttons. To test implicit knowledge transfer, the researchers relied on "demonstration points"—newcomers could observe the collective's current voting patterns and actions to infer the correct strategy.

Results

Individual Learning and Efficiency

Workers showed a clear ability to learn and remember task solutions. In the door-opening experiment, the average number of attempts required to find the target dropped significantly after the first trial.

Figure 1: Average number of doors opened across three consecutive attempts, showing rapid convergence toward the optimal solution.

As seen in Figure 1, workers moved from a median of 12.16 clicks to just 3.64 by their third attempt. Remarkably, workers returning after a 6-hour break retained 99% of this performance improvement, proving that crowd workers are not merely "cogs" but learners who retain task-specific expertise.

Collective Accuracy and Turnover

The crowd demonstrated a robust ability to maintain accuracy as a single agent. In the continuous navigation task, the collective reached and maintained high accuracy levels despite complete turnovers of the worker pool.

Figure 3: Collective accuracy over multiple rounds of the navigation task, demonstrating the crowd's ability to reach 100% accuracy through shared learning.

As shown in Figure 3, the percentage of correct actions reached 100% by the fifth round. This "collective memory" is maintained through the overlap of worker generations; as long as some experienced workers remain to demonstrate the correct action, new workers can learn the strategy implicitly.

Discussion and Limitations

The study concludes that a crowd can be more reliable than a single expert because it can remain active indefinitely through a recruitment pipeline. However, this collective memory is fragile. If the number of active workers drops to zero, the "chain of demonstration" is broken, and the memory is lost.

Future work could enhance this memory by introducing explicit communication channels, such as history logs or chat systems, allowing the crowd to handle more nuanced, non-repetitive tasks. Additionally, while individual retention is high, current marketplaces are not optimized for re-recruiting specific experienced workers, representing a missed opportunity for efficiency in large-scale crowd systems.

Summary updated

Abstract

Crowd algorithms often assume workers are inexperienced and thus fail to adapt as workers in the crowd learn a task. These assumptions fundamentally limit the types of tasks that systems based on such algorithms can handle. This paper explores how the crowd learns and remembers over time in the context of human computation, and how more realistic assumptions of worker experience may be used when designing new systems. We first demonstrate that the crowd can recall information over time and discuss possible implications of crowd memory in the design of crowd algorithms. We then explore crowd learning during a continuous control task. Recent systems are able to disguise dynamic groups of workers as crowd agents to support continuous tasks, but have not yet considered how such agents are able to learn over time. We show, using a real-time gaming setting, that crowd agents can learn over time, and `remember' by passing strategies from one generation of workers to the next, despite high turnover rates in the workers comprising them. We conclude with a discussion of future research directions for crowd memory and learning.
BibTeX
@misc{lasecki2012crowd,
  author = {Walter S. Lasecki and Samuel White and Kyle I. Murray and Jeffrey P. Bigham},
  title = {{Crowd Memory: Learning in the Collective}},
  year = {2012},
  eprint = {1204.3678},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1204.3678},
}
Figure