The Pushshift Reddit Dataset

Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, Jeremy Blackburn

TL;DR

The paper introduces the Pushshift Reddit Dataset, a comprehensive, real-time archive of public Reddit activity dating back to the platform's inception in 2005. By combining monthly bulk dumps, a flexible REST API, and interactive community tooling, Pushshift removes the massive engineering overhead typically required to collect and analyze social data at web scale.

What Problem Does This Paper Address?

We are currently in a "post-API age": major platforms like Facebook and Twitter have aggressively restricted their public data interfaces following high-profile privacy scandals. While Reddit remained comparatively accessible, researchers still faced steep technical and logistical barriers when attempting to gather historical or platform-wide data. Collecting billions of interactions requires substantial scraping pipelines, large storage capacity, and significant computational overhead—often turning social computing research into an exhausting data engineering chore. Furthermore, Reddit's official API limits individual queries to 100 objects and lacks built-in aggregation or full-text search across historical records.

Methods

Pushshift solves this with an end-to-end ingest, storage, and querying pipeline:

  • Ingest Engine: Language-agnostic worker programs pull raw Reddit content via APIs and web streams, pushing items into a Redis-backed staging queue for processing and normalization before flushing raw newline-delimited JSON (ndjson) to disk.
  • Queryable Document Store & DB: Ingested objects are routed to an Elasticsearch cluster for fast indexing, full-text search, and aggregation, alongside a PostgreSQL database for relational metadata storage. Dynamic mappings adapt on the fly to Reddit's unversioned API changes, while an ICU plugin enables multilingual, Unicode 12, and emoji search.
  • Access Endpoints: Users can query the platform through a REST API (with batch limits 5× larger than Reddit’s native API) or interact with real-time Slackbots to generate exploratory visualizations directly in chat.
  • FAIR Compliance: Data dumps are made freely available in standardized ndjson formats. To support academic permanence, a sample subset is registered with a persistent DOI on Zenodo.

Datasets

The primary contribution is the dataset itself:

  • Scope: Complete public Reddit submissions and comments from June 2005 through April 2019 (and continuously updating in real-time).
  • Scale: 651,778,198 submissions and 5,601,331,385 comments spanning 2,888,885 subreddits.
  • Data Schema: Each entry is an annotated JSON record containing post/comment identifiers, author handles, timestamps (created_utc), parent/link trees, full text (selftext or body), scores, moderation tags (distinguished, quarantine, stickied), and subreddit-specific flair.

Key Insights and Ideas

  • Datasets as Infrastructure: Providing pre-collected, historical datasets levels the playing field for researchers lacking massive cloud budgets, counteracting the growing stratification of academic research.
  • Search Over Raw Dumps: Most researchers do not need multiple terabytes of raw data on local disk. Providing an aggregation-capable Elasticsearch API (handling over 500 million requests monthly) drastically cuts down exploratory research time.
  • Ecosystem Agility: Because Reddit frequently alters its JSON structures without versioning notices, building ingest architectures with dynamic schema mapping is essential for long-term longitudinal scraping.

Key Results

  • Data Velocity: Growth curves show comments climbing past 1 million per day after August 2013 and reaching 5 million comments per day by April 2019. Submissions stabilized at over 500,000 per day.
  • Community Adoption: By late 2019, over 100 peer-reviewed papers had leveraged Pushshift data across diverse domains, including online community governance (e.g., studying the impact of banning toxic subreddits), political extremism, disinformation campaigns, clinical NLP, and health informatics.
  • Active Community: The project maintains an active user base, including a 2,100+ member subreddit (/r/pushshift) and a specialized Slack team generating hundreds of thousands of analytical discussions.

Limitations and Future Work

  • Data Completeness & Validation: Prior scrutiny (e.g., Gaffney & Matias 2018) highlighted gaps and missing records in earlier Pushshift archives, prompting continuous validation efforts.
  • Archive Size vs. Open Repositories: The complete corpus spans several terabytes, exceeding the limits of traditional academic archives (e.g., Zenodo’s 100GB ceiling), necessitating independent hosting and maintenance by Pushshift.
  • Platform Volatility: Pushshift remains dependent on upstream API stability and terms from Reddit, meaning changes to Reddit's underlying systems require constant maintenance.

Summary updated

Abstract

Social media data has become crucial to the advancement of scientific understanding. However, even though it has become ubiquitous, just collecting large-scale social media data involves a high degree of engineering skill set and computational resources. In fact, research is often times gated by data engineering problems that must be overcome before analysis can proceed. This has resulted recognition of datasets as meaningful research contributions in and of themselves.Reddit, the so called “front page of the Internet,” in particular has been the subject of numerous scientific studies. Although Reddit is relatively open to data acquisition compared to social media platforms like Facebook and Twitter, the technical barriers to acquisition still remain. Thus, Reddit's millions of subreddits, hundreds of millions of users, and billions of comments are at the same time relatively accessible, but time consuming to collect and analyze systematically.In this paper, we present the Pushshift Reddit dataset. Pushshift is a social media data collection, analysis, and archiving platform that since 2015 has collected Reddit data and made it available to researchers. Pushshift's Reddit dataset is updated in real-time, and includes historical data back to Reddit's inception. In addition to monthly dumps, Pushshift provides computational tools to aid in searching, aggregating, and performing exploratory analysis on the entirety of the dataset. The Pushshift Reddit dataset makes it possible for social media researchers to reduce time spent in the data collection, cleaning, and storage phases of their projects.
BibTeX
@inproceedings{baumgartner2020pushshift,
  author = {Jason Baumgartner and Savvas Zannettou and Brian Keegan and Megan Squire and Jeremy Blackburn},
  title = {{The Pushshift Reddit Dataset}},
  booktitle = {ICWSM},
  year = {2020},
  url = {https://openreview.net/forum?id=Uud7rrdBtfo},
}

Loading related content...

Figure