The United States COVID-19 Forecast Hub dataset

Estee Y. Cramer, Yuxin Huang, Yijin Wang, Evan L Ray, Matthew Cornell, Johannes Bracher, Andrea Brennen, Alvaro J Castero Rivadeneira, Aaron Gerding, Katie House, Jimeng Sun, Abdul Hannan Kanji, Ayush Khandelwal, Khoa Le, Jarad Niemi, Ariane Stark, Apurv Shah, Nutcha Wattanchit, Martha Zorn, Nicholas G ReichmedRxiv 2021 · DOI

Abstract

During the COVID-19 pandemic, academic researchers, government agencies, and industry groups produced forecasts at an unprecedented scale. However, the lack of standardization across these models initially hindered their utility for public health decision-making. To address this, the United States Centers for Disease Control and Prevention (CDC) partnered with the University of Massachusetts Amherst to launch the US COVID-19 Forecast Hub in April 2020. This initiative established a centralized, open-source repository for point and probabilistic forecasts of COVID-19 cases, hospitalizations, and deaths. By providing a standardized framework for data submission and validation, the Hub enabled the creation of robust ensemble models that have informed national policy and public health communication. This paper details the infrastructure, methodology, and impact of this collaborative dataset.

The Challenge of Fragmented Forecasting

Early in the COVID-19 pandemic, the modeling community rushed to provide predictions to guide the public health response. However, these efforts were largely siloed. Different groups used disparate data sources, varying time horizons, and inconsistent file formats. This fragmentation made it nearly impossible for public health officials to compare models, evaluate their accuracy in real-time, or combine them into a more robust "consensus" view.

The US COVID-19 Forecast Hub was created to solve this coordination problem. By establishing a centralized infrastructure, the Hub provided a standardized, comparable set of short-term forecasts, allowing for the development of ensemble models that generally outperform any single individual model.

Technical Infrastructure and Methodology

The Hub operates as a sophisticated data pipeline designed for "prospective" validation—ensuring forecasts are recorded before the events they predict actually occur. The workflow consists of several key technical components:

  • Standardized Submission: Modeling teams submit their forecasts as CSV files via GitHub pull requests. This utilizes a familiar version-control environment to manage contributions from over 82 different teams.
  • Automated Validation: Every submission undergoes automated checks via GitHub Actions. These scripts ensure the data meets strict formatting rules, such as correct date formats and the inclusion of required probability quantiles.
  • Data Backend: Validated forecasts are synchronized with Zoltar, a specialized forecast repository database. Zoltar provides a REST API that allows researchers to query and extract specific subsets of the massive dataset easily.
  • Model Designations: To maintain the integrity of official reports, models are categorized as "primary," "secondary," or "other." Only primary models—representing a team's main forecasting effort—are typically included in the official CDC ensemble.

The Meta-Dataset and Ground Truth

The Forecast Hub is essentially a meta-dataset of predictions. To evaluate the performance of these predictions, the Hub integrates several "ground truth" sources to represent the actual observed data:

  • JHU CSSE: The primary source for reported cases and deaths.
  • HealthData.gov (HHS): The source for state-level hospitalization data.
  • USA Facts & NYTimes: Used as secondary or alternative ground truth sources for cross-validation.

The resulting dataset contains point and probabilistic (quantile-based) forecasts for incident cases, hospitalizations, and deaths at the national, state, and county levels across the United States.

The Power of Standardization and Ensembles

A core insight from the Hub project is that technical standardization—the "boring" work of aligning CSV columns and enforcing data schemas—is what enables high-level science and policy decisions. By requiring specific quantiles, the Hub allowed for the creation of a "weighted average" ensemble model.

This ensemble approach proved that combining diverse modeling strategies (ranging from mechanistic SEIR models to deep learning approaches) provides a more reliable signal than relying on any single "genius" model. By time-stamping every submission on GitHub, the Hub also created a public, independent registration system that prevents "hindsight bias," where modelers might be tempted to tweak results after seeing the actual data.

Results and Impact

As of September 2021, the Forecast Hub has achieved significant scale and influence:

  • Data Volume: The dataset contains nearly 65 million individual predictions within more than 4,600 submitted files.
  • Participation: Over 100 unique models from more than 82 modeling teams have contributed.
  • Consistency: During the first half of 2021, a median of 35 to 44 teams contributed forecasts every single week.
  • Policy Integration: The Hub’s ensemble model became the primary forecasting tool used in the CDC’s weekly public briefings, providing the official outlook for the pandemic's trajectory in the U.S.

Limitations and Future Directions

Despite its success, the authors acknowledge several technical constraints. The quantile format used for submissions, while efficient, does not perfectly capture "tail risks" or extreme outliers. Furthermore, the current format does not preserve information regarding how different outcomes (e.g., the relationship between a spike in cases and a subsequent spike in deaths) are correlated within a single model. Finally, due to the sheer volume of data at the county level (over 3,000 locations), the Hub had to accept "coarser" case forecasts with fewer quantiles to maintain system performance.

However, the "Hub" structure has proven to be a successful blueprint for open-science collaboration. It is already being adapted for use in the European Union and the United Kingdom, and it serves as a foundational model for tracking future infectious disease outbreaks, such as seasonal influenza or future pandemics.

Summary updated

Abstract

Abstract Academic researchers, government agencies, industry groups, and individuals have produced forecasts at an unprecedented scale during the COVID-19 pandemic. To leverage these forecasts, the United States Centers for Disease Control and Prevention (CDC) partnered with an academic research lab at the University of Massachusetts Amherst to create the US COVID-19 Forecast Hub. Launched in April 2020, the Forecast Hub is a dataset with point and probabilistic forecasts of incident hospitalizations, incident cases, incident deaths, and cumulative deaths due to COVID-19 at national, state, and county levels in the United States. Included forecasts represent a variety of modeling approaches, data sources, and assumptions regarding the spread of COVID-19. The goal of this dataset is to establish a standardized and comparable set of short-term forecasts from modeling teams. These data can be used to develop ensemble models, communicate forecasts to the public, create visualizations, compare models, and inform policies regarding COVID-19 mitigation. These open-source data are available via download from GitHub, through an online API, and through R packages.
BibTeX
@misc{cramer2021united,
  author = {Estee Y. Cramer and Yuxin Huang and Yijin Wang and Evan L Ray and Matthew Cornell and Johannes Bracher and Andrea Brennen and Alvaro J Castero Rivadeneira and Aaron Gerding and Katie House and Jimeng Sun and Abdul Hannan Kanji and Ayush Khandelwal and Khoa Le and Jarad Niemi and Ariane Stark and Apurv Shah and Nutcha Wattanchit and Martha Zorn and Nicholas G Reich},
  title = {{The United States COVID-19 Forecast Hub dataset}},
  year = {2021},
  doi = {10.1101/2021.11.04.21265886},
  howpublished = {medRxiv},
}
Figure