Abstract
During the COVID-19 pandemic, academic researchers, government agencies, and industry groups produced forecasts at an unprecedented scale. However, the lack of standardization across these models initially hindered their utility for public health decision-making. To address this, the United States Centers for Disease Control and Prevention (CDC) partnered with the University of Massachusetts Amherst to launch the US COVID-19 Forecast Hub in April 2020. This initiative established a centralized, open-source repository for point and probabilistic forecasts of COVID-19 cases, hospitalizations, and deaths. By providing a standardized framework for data submission and validation, the Hub enabled the creation of robust ensemble models that have informed national policy and public health communication. This paper details the infrastructure, methodology, and impact of this collaborative dataset.
The Challenge of Fragmented Forecasting
Early in the COVID-19 pandemic, the modeling community rushed to provide predictions to guide the public health response. However, these efforts were largely siloed. Different groups used disparate data sources, varying time horizons, and inconsistent file formats. This fragmentation made it nearly impossible for public health officials to compare models, evaluate their accuracy in real-time, or combine them into a more robust "consensus" view.
The US COVID-19 Forecast Hub was created to solve this coordination problem. By establishing a centralized infrastructure, the Hub provided a standardized, comparable set of short-term forecasts, allowing for the development of ensemble models that generally outperform any single individual model.
Technical Infrastructure and Methodology
The Hub operates as a sophisticated data pipeline designed for "prospective" validation—ensuring forecasts are recorded before the events they predict actually occur. The workflow consists of several key technical components:
- Standardized Submission: Modeling teams submit their forecasts as CSV files via GitHub pull requests. This utilizes a familiar version-control environment to manage contributions from over 82 different teams.
- Automated Validation: Every submission undergoes automated checks via GitHub Actions. These scripts ensure the data meets strict formatting rules, such as correct date formats and the inclusion of required probability quantiles.
- Data Backend: Validated forecasts are synchronized with Zoltar, a specialized forecast repository database. Zoltar provides a REST API that allows researchers to query and extract specific subsets of the massive dataset easily.
- Model Designations: To maintain the integrity of official reports, models are categorized as "primary," "secondary," or "other." Only primary models—representing a team's main forecasting effort—are typically included in the official CDC ensemble.
The Meta-Dataset and Ground Truth
The Forecast Hub is essentially a meta-dataset of predictions. To evaluate the performance of these predictions, the Hub integrates several "ground truth" sources to represent the actual observed data:
- JHU CSSE: The primary source for reported cases and deaths.
- HealthData.gov (HHS): The source for state-level hospitalization data.
- USA Facts & NYTimes: Used as secondary or alternative ground truth sources for cross-validation.
The resulting dataset contains point and probabilistic (quantile-based) forecasts for incident cases, hospitalizations, and deaths at the national, state, and county levels across the United States.
The Power of Standardization and Ensembles
A core insight from the Hub project is that technical standardization—the "boring" work of aligning CSV columns and enforcing data schemas—is what enables high-level science and policy decisions. By requiring specific quantiles, the Hub allowed for the creation of a "weighted average" ensemble model.
This ensemble approach proved that combining diverse modeling strategies (ranging from mechanistic SEIR models to deep learning approaches) provides a more reliable signal than relying on any single "genius" model. By time-stamping every submission on GitHub, the Hub also created a public, independent registration system that prevents "hindsight bias," where modelers might be tempted to tweak results after seeing the actual data.
Results and Impact
As of September 2021, the Forecast Hub has achieved significant scale and influence:
- Data Volume: The dataset contains nearly 65 million individual predictions within more than 4,600 submitted files.
- Participation: Over 100 unique models from more than 82 modeling teams have contributed.
- Consistency: During the first half of 2021, a median of 35 to 44 teams contributed forecasts every single week.
- Policy Integration: The Hub’s ensemble model became the primary forecasting tool used in the CDC’s weekly public briefings, providing the official outlook for the pandemic's trajectory in the U.S.
Limitations and Future Directions
Despite its success, the authors acknowledge several technical constraints. The quantile format used for submissions, while efficient, does not perfectly capture "tail risks" or extreme outliers. Furthermore, the current format does not preserve information regarding how different outcomes (e.g., the relationship between a spike in cases and a subsequent spike in deaths) are correlated within a single model. Finally, due to the sheer volume of data at the county level (over 3,000 locations), the Hub had to accept "coarser" case forecasts with fewer quantiles to maintain system performance.
However, the "Hub" structure has proven to be a successful blueprint for open-science collaboration. It is already being adapted for use in the European Union and the United Kingdom, and it serves as a foundational model for tracking future infectious disease outbreaks, such as seasonal influenza or future pandemics.
Summary updated