CrowdPlay: Crowdsourcing Human Demonstrations for Offline Learning

Matthias Gerstgrasser, Rakshit Trivedi, David C. ParkesICLR 2022 · OpenReview

In fields like computer vision and natural language processing, massive datasets—such as ImageNet or Common Crawl—have fueled the deep learning revolution. Reinforcement Learning (RL), however, has largely relied on "online" interaction with simulators or "offline" datasets generated by synthetic AI agents.

The problem is that synthetic data is often "too clean." It fails to capture the sub-optimal, diverse, and multimodal nature of real human behavior. Collecting human data at scale is traditionally difficult, expensive, and prone to low quality. To address this gap, researchers have introduced CrowdPlay, a framework designed to bring the "ImageNet moment" to RL by crowdsourcing large-scale human demonstrations.

The CrowdPlay Platform

CrowdPlay is a highly scalable client-server architecture built to collect high-quality gameplay data efficiently.

CrowdPlay Architecture

The platform features a React-based frontend that allows users to play games directly in their browser, while a Python backend interfaces with standard RL environments like OpenAI Gym. It handles everything from keypress logging to trajectory compression.

Key features of the platform include:

  • Multi-Agent Support: Unlike many previous platforms, CrowdPlay supports human-human and human-AI interactions, including "fallback" AI agents that take over if a human player disconnects.
  • Incentive Engineering: The system incorporates real-time feedback. For paid workers (like those on Amazon Mechanical Turk), it displays live progress toward bonuses based on specific behavioral goals. For unpaid volunteers, it uses high score tables and challenges to drive engagement.
  • Multimodal Instructions: To ensure data diversity, researchers gave players specific behavioral instructions (e.g., "only stay on the left side of the screen" or "shoot aliens row-by-row") rather than simply asking them to "play well."

The CrowdPlay Atari Dataset

Using this platform, the authors compiled a massive dataset comprising over 250 hours of gameplay and 54 million transitions across six Atari 2600 classics (including Space Invaders, River Raid, and Montezuma’s Revenge).

Data was collected from a diverse pool of participants: Amazon Mechanical Turk workers (paid), university students (raffle-based), and social media users (voluntary). Notably, it is the first Atari dataset to feature explicitly multimodal behavior and multi-agent data, encompassing both competitive and cooperative play.

Key Insights: The Reality of Human Data

Analyzing this dataset revealed several sobering truths about the current state of offline RL when faced with the complexity of real human behavior.

Demographics Matter "Human data" is not a monolith. t-SNE embeddings of action distributions showed clear clusters based on the recruitment channel. MTurk workers, students, and social media users exhibited distinctly different playstyles.

t-SNE Clusters by Recruitment Channel

The "Human Gap" in Offline RL Algorithms like Conservative Q-Learning (CQL) and Batch Constrained Q-Learning (BCQ), which perform excellently on synthetic data, struggled significantly with this human dataset. They often failed to reach even 45% of the performance of the best human demonstrators. Standard DQN actually served as a surprisingly robust offline baseline, often outperforming more complex "conservative" algorithms designed specifically for offline learning. The Implicit Quantile Network (IQN) generally performed the best among tested algorithms, but still fell short of human experts in most tasks.

Multimodality is a Hurdle While human players easily switched between different "modes" of play based on instructions, standard Behavior Cloning (BC) agents failed to capture these distinct clusters. This often resulted in a "blurry" average policy that performed poorly.

Incentives are the "Reward Function" of Crowdsourcing The researchers found that simply paying for "active time" resulted in low-quality data. However, tying payments to specific performance metrics (quality-based incentives) significantly boosted adherence to instructions. While social media recruitment yielded the highest quality data per user, MTurk was the only channel capable of scaling to hundreds of hours of data quickly.

Looking Forward

The CrowdPlay framework and dataset highlight a dire need for offline RL algorithms that are robust to the noisy, sub-optimal, and multimodal nature of real-world human data. By providing the infrastructure to collect this data and a benchmark to test against it, CrowdPlay takes a crucial step toward effective RL in real-world settings.

Summary updated

Abstract

Crowdsourcing has been instrumental for driving AI advances that rely on large-scale data. At the same time, reinforcement learning has seen rapid progress through benchmark environments that strike a balance between tractability and real-world complexity, such as ALE and OpenAI Gym. In this paper, we aim to fill a gap at the intersection of these two: The use of crowdsourcing to generate large-scale human demonstration data in the support of advancing research into imitation learning and offline learning. To this end, we present CrowdPlay, a complete crowdsourcing pipeline for any standard RL environment including OpenAI Gym (made available under an open-source license); a large-scale publicly available crowdsourced dataset of human gameplay demonstrations in Atari 2600 games, including multimodal behavior and human-human and human-AI multiagent data; offline learning benchmarks with extensive human data evaluation; and a detailed study of incentives, including real-time feedback to drive high quality data. We hope that this will drive the improvement in design of algorithms that account for the complexity of human, behavioral data and thereby enable a step forward in direction of effective learning for real-world settings. Our code and dataset are available at https://mgerstgrasser.github.io/crowdplay/.
BibTeX
@inproceedings{gerstgrasser2022crowdplay,
  author = {Matthias Gerstgrasser and Rakshit Trivedi and David C. Parkes},
  title = {{CrowdPlay: Crowdsourcing Human Demonstrations for Offline Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=qyTBxTztIpQ},
}
Figure