Abstract: As the COVID-19 outbreak evolves, accurate forecasting continues to play an extremely important role in informing policy decisions. In this paper, we present our continuous curation of a large data repository containing COVID-19 information from a range of sources. We use this data to develop predictions and corresponding prediction intervals for the short-term trajectory of COVID-19 cumulative death counts at the county-level in the United States up to two weeks ahead. Using data from January 22 to June 20, 2020, we develop and combine multiple forecasts using ensembling techniques, resulting in an ensemble we refer to as Combined Linear and Exponential Predictors (CLEP). Our individual predictors include county-specific exponential and linear predictors, a shared exponential predictor that pools data together across counties, an expanded shared exponential predictor that uses data from neighboring counties, and a demographics-based shared exponential predictor. We use prediction errors from the past five days to assess the uncertainty of our death predictions, resulting in generally-applicable prediction intervals, Maximum (absolute) Error Prediction Intervals (MEPI). MEPI achieves a coverage rate of more than 94% when averaged across counties for predicting cumulative recorded death counts two weeks in the future. Our forecasts are currently being used by the non-profit organization, Response4Life, to determine the medical supply need for individual hospitals and have directly contributed to the distribution of medical supplies across the country. We hope that our forecasts and data repository at https://covidseverity.com can help guide necessary county-specific decision-making and help counties prepare for their continued fight against COVID-19.
Introduction
In the early months of the COVID-19 pandemic, most forecasting models focused on national or state-level trends. However, critical policy decisions—such as closing schools, implementing stay-at-home orders, or allocating personal protective equipment (PPE)—occur primarily at the local level. Predicting mortality at the county level is notoriously difficult due to "noisy" data from reporting lags, limited testing availability, and the heterogeneous nature of the virus's spread across urban and rural environments. Furthermore, many existing models provide prediction intervals that are too narrow, failing to capture observed deaths and leading to a false sense of certainty.
Data Curation and Repository
To address these challenges, we curated a massive open-source repository containing over 8,000 features. This repository serves as the foundation for our localized forecasting efforts. Key data components include:
- COVID-19 Statistics: Daily cases and deaths sourced from USAFacts and the New York Times.
- County Features: Census demographics, socioeconomic factors from the USDA, and health risk indicators from the CDC.
- Healthcare Infrastructure: Data on over 7,000 hospitals, including ICU bed counts and staffing levels from HRSA and the Kaiser Family Foundation.
- Mobility Data: Anonymized movement trends provided by Google and Apple.
We chose to forecast deaths rather than cases because death counts—while still subject to reporting anomalies—serve as a more stable proxy for the true severity of the outbreak, as they are less affected by fluctuating testing capacity.
The CLEP Ensemble
Instead of relying on a single model, we developed the Combined Linear and Exponential Predictors (CLEP) ensemble. This approach combines five distinct predictors to capture the varied growth trajectories observed across different counties:
- Separate-county exponential and linear predictors: Individual fits for each county.
- Shared-county exponential predictor: Pools data across the nation to learn general trends.
- Expanded shared exponential predictor: Incorporates data from neighboring counties.
- Demographics-based shared exponential predictor: Groups counties with similar demographic profiles.
CLEP utilizes an adaptive weighting scheme that gives more weight to predictors that performed well over the previous week. By including a linear predictor, the ensemble can effectively "pivot" when a county begins to flatten its curve, moving away from purely exponential assumptions.
Uncertainty Quantification via MEPI
To quantify uncertainty without relying on traditional probabilistic assumptions that often fail during a dynamic crisis, we developed Maximum-absolute-Error Prediction Intervals (MEPI). Inspired by conformal inference, MEPI calculates the maximum normalized error the model made over the last five days and uses that to "pad" the future forecast.
We found that a short, 5-day rolling window is optimal because the pandemic is highly dynamic; errors from two weeks prior are often "stale" and do not reflect current trends. This data-driven approach ensures the intervals remain responsive to immediate shifts in the pandemic's trajectory.
Results and Real-World Impact
Our evaluation using data from March 22 to June 20, 2020, demonstrates the robustness of our approach:
- Accuracy: The CLEP model achieved a Median Absolute Percentage Error (MAPE) of approximately 15% for 7-day forecasts and 26% for 14-day forecasts.
- Reliability: The MEPI intervals achieved a coverage rate of over 94% on average. Even in hard-hit counties (those with 10+ deaths), coverage remained near 90%, significantly outperforming contemporary models.
Beyond academic metrics, these forecasts have had a direct real-world impact. Our results were used by the non-profit organization Response4Life to prioritize the delivery of medical supplies to hospitals most likely to face a surge in mortality. By providing reliable, county-level insights, this work helps local officials and healthcare providers prepare for the continued challenges of the COVID-19 pandemic.
Summary updated