The Pushshift Reddit Dataset

Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, Jeremy Blackburn

Abstract

Social media data has become crucial to the advancement of scientific understanding. However, even though it has become ubiquitous, just collecting large-scale social media data involves a high degree of engineering skill set and computational resources. In fact, research is often times gated by data engineering problems that must be overcome before analysis can proceed. This has resulted in the recognition of datasets as meaningful research contributions in and of themselves. Reddit, the so-called "front page of the Internet," in particular has been the subject of numerous scientific studies. Although Reddit is relatively open to data acquisition compared to social media platforms like Facebook and Twitter, the technical barriers to acquisition still remain. Thus, Reddit's millions of subreddits, hundreds of millions of users, and hundreds of billions of comments are at the same time relatively accessible, but time-consuming to collect and analyze systematically. In this paper, we present the Pushshift Reddit dataset. Pushshift is a social media data collection, analysis, and archiving platform that since 2015 has collected Reddit data and made it available to researchers. Pushshift's Reddit dataset is updated in real-time and includes historical data back to Reddit's inception. In addition to monthly dumps, Pushshift provides computational tools to aid in searching, aggregating, and performing exploratory analysis on the entirety of the dataset. The Pushshift Reddit dataset makes it possible for social media researchers to reduce time spent in the data collection, cleaning, and storage phases of their projects.

Introduction: The Data Engineering Barrier

We are currently living in what researchers call the "post-API age." Following major privacy scandals, dominant social media platforms like Facebook and Twitter have severely restricted their public APIs. This has created a "chilling effect" on social science, where data access is often gated by technical expertise, financial capital, or "trusted partner" status. While Reddit remains relatively open, the sheer scale of the platform—millions of subreddits and hundreds of billions of comments—makes systematic collection a daunting engineering task for most researchers. The Pushshift project addresses this by handling the "heavy lifting" of data ingestion, cleaning, and storage, allowing researchers to focus on analysis rather than data engineering.

System Architecture

The Pushshift platform utilizes a robust backend architecture designed to handle the continuous "firehose" of Reddit activity. The system is composed of several key layers:

  • Ingest Engine: A language-agnostic framework that crawls and scrapes Reddit data, using Redis as an intermediate staging area to queue data before processing.
  • Storage & Indexing: Data is stored in a PostgreSQL database for metadata management and an Elasticsearch (ES) cluster for high-performance querying and full-text search.
  • Dynamic Mapping: Because Reddit’s official API frequently changes its schema without versioning, Pushshift utilizes Elasticsearch’s dynamic mapping to automatically accommodate new data fields as they appear.
System architecture of the Pushshift platform, showing the flow from Reddit to ingest and queryable data stores.

Dataset Characteristics and Growth

The Pushshift Reddit Dataset provides a comprehensive historical record of the platform. At the time of publication, the dataset included over 651 million submissions and 5.6 billion comments, spanning from Reddit's inception in 2005 through April 2019. This data covers over 2.8 million subreddits and is provided in newline-delimited JSON (ndjson) format.

One of the primary values of the dataset is its ability to quantify Reddit's explosive growth. As shown in the figure below, the platform's activity has scaled significantly over the last decade. In August 2013, the platform saw approximately 1 million comments per day; by April 2019, that volume had quintupled to 5 million comments per day.

Growth of Reddit submissions and comments from 2005 to early 2019, illustrating the massive scale of the dataset.

Impact and Adoption

By providing a "social infrastructure" for data, Pushshift democratizes research, allowing smaller teams to compete with well-funded labs. As of late 2019, over 100 peer-reviewed publications had already utilized Pushshift data across fields like health informatics, political science, and natural language processing (NLP). The project also maintains an active community of over 2,100 subscribers on its own subreddit and a Slack team of 300+ researchers, facilitating collaborative data validation.

Ethical and Technical Challenges

Despite its success, the project faces ongoing challenges. The sheer size of the dataset (multiple terabytes) exceeds the limits of free repositories like Zenodo, making long-term, decentralized archiving difficult. Furthermore, as privacy norms shift, the project must navigate the tension between providing a permanent public record for science and respecting the "right to be forgotten" or platform-level deletions. The authors continue to improve ingestion pipelines to address data gaps and maintain transparency regarding the dataset's integrity.

Summary updated

Abstract

Social media data has become crucial to the advancement of scientific understanding. However, even though it has become ubiquitous, just collecting large-scale social media data involves a high degree of engineering skill set and computational resources. In fact, research is often times gated by data engineering problems that must be overcome before analysis can proceed. This has resulted recognition of datasets as meaningful research contributions in and of themselves. Reddit, the so called "front page of the Internet," in particular has been the subject of numerous scientific studies. Although Reddit is relatively open to data acquisition compared to social media platforms like Facebook and Twitter, the technical barriers to acquisition still remain. Thus, Reddit's millions of subreddits, hundreds of millions of users, and hundreds of billions of comments are at the same time relatively accessible, but time consuming to collect and analyze systematically. In this paper, we present the Pushshift Reddit dataset. Pushshift is a social media data collection, analysis, and archiving platform that since 2015 has collected Reddit data and made it available to researchers. Pushshift's Reddit dataset is updated in real-time, and includes historical data back to Reddit's inception. In addition to monthly dumps, Pushshift provides computational tools to aid in searching, aggregating, and performing exploratory analysis on the entirety of the dataset. The Pushshift Reddit dataset makes it possible for social media researchers to reduce time spent in the data collection, cleaning, and storage phases of their projects.
BibTeX
@inproceedings{baumgartner2020pushshift,
  author = {Jason Baumgartner and Savvas Zannettou and Brian Keegan and Megan Squire and Jeremy Blackburn},
  title = {{The Pushshift Reddit Dataset}},
  booktitle = {ICWSM},
  year = {2020},
  url = {https://openreview.net/forum?id=Uud7rrdBtfo},
}
Figure