TL;DR
The paper introduces the Pushshift Reddit Dataset, a comprehensive, real-time archive of public Reddit activity dating back to the platform's inception in 2005. By combining monthly bulk dumps, a flexible REST API, and interactive community tooling, Pushshift removes the massive engineering overhead typically required to collect and analyze social data at web scale.
What Problem Does This Paper Address?
We are currently in a "post-API age": major platforms like Facebook and Twitter have aggressively restricted their public data interfaces following high-profile privacy scandals. While Reddit remained comparatively accessible, researchers still faced steep technical and logistical barriers when attempting to gather historical or platform-wide data. Collecting billions of interactions requires substantial scraping pipelines, large storage capacity, and significant computational overhead—often turning social computing research into an exhausting data engineering chore. Furthermore, Reddit's official API limits individual queries to 100 objects and lacks built-in aggregation or full-text search across historical records.
Methods
Pushshift solves this with an end-to-end ingest, storage, and querying pipeline:
- Ingest Engine: Language-agnostic worker programs pull raw Reddit content via APIs and web streams, pushing items into a Redis-backed staging queue for processing and normalization before flushing raw newline-delimited JSON (
ndjson) to disk. - Queryable Document Store & DB: Ingested objects are routed to an Elasticsearch cluster for fast indexing, full-text search, and aggregation, alongside a PostgreSQL database for relational metadata storage. Dynamic mappings adapt on the fly to Reddit's unversioned API changes, while an ICU plugin enables multilingual, Unicode 12, and emoji search.
- Access Endpoints: Users can query the platform through a REST API (with batch limits 5× larger than Reddit’s native API) or interact with real-time Slackbots to generate exploratory visualizations directly in chat.
- FAIR Compliance: Data dumps are made freely available in standardized
ndjsonformats. To support academic permanence, a sample subset is registered with a persistent DOI on Zenodo.
Datasets
The primary contribution is the dataset itself:
- Scope: Complete public Reddit submissions and comments from June 2005 through April 2019 (and continuously updating in real-time).
- Scale: 651,778,198 submissions and 5,601,331,385 comments spanning 2,888,885 subreddits.
- Data Schema: Each entry is an annotated JSON record containing post/comment identifiers, author handles, timestamps (
created_utc), parent/link trees, full text (selftextorbody), scores, moderation tags (distinguished,quarantine,stickied), and subreddit-specific flair.
Key Insights and Ideas
- Datasets as Infrastructure: Providing pre-collected, historical datasets levels the playing field for researchers lacking massive cloud budgets, counteracting the growing stratification of academic research.
- Search Over Raw Dumps: Most researchers do not need multiple terabytes of raw data on local disk. Providing an aggregation-capable Elasticsearch API (handling over 500 million requests monthly) drastically cuts down exploratory research time.
- Ecosystem Agility: Because Reddit frequently alters its JSON structures without versioning notices, building ingest architectures with dynamic schema mapping is essential for long-term longitudinal scraping.
Key Results
- Data Velocity: Growth curves show comments climbing past 1 million per day after August 2013 and reaching 5 million comments per day by April 2019. Submissions stabilized at over 500,000 per day.
- Community Adoption: By late 2019, over 100 peer-reviewed papers had leveraged Pushshift data across diverse domains, including online community governance (e.g., studying the impact of banning toxic subreddits), political extremism, disinformation campaigns, clinical NLP, and health informatics.
- Active Community: The project maintains an active user base, including a 2,100+ member subreddit (
/r/pushshift) and a specialized Slack team generating hundreds of thousands of analytical discussions.
Limitations and Future Work
- Data Completeness & Validation: Prior scrutiny (e.g., Gaffney & Matias 2018) highlighted gaps and missing records in earlier Pushshift archives, prompting continuous validation efforts.
- Archive Size vs. Open Repositories: The complete corpus spans several terabytes, exceeding the limits of traditional academic archives (e.g., Zenodo’s 100GB ceiling), necessitating independent hosting and maintenance by Pushshift.
- Platform Volatility: Pushshift remains dependent on upstream API stability and terms from Reddit, meaning changes to Reddit's underlying systems require constant maintenance.
Summary updated