Abstract
Predicting the trajectory of an epidemic beyond a few weeks is inherently limited by the complexity of disease systems and the unpredictability of human behavior. While short-term forecasts rely on current momentum, long-term projections require a different approach. The U.S. COVID-19 Scenario Modeling Hub (SMH) was established to provide 6-month-ahead projections by defining specific "what-if" scenarios regarding vaccine uptake and variant transmissibility. This paper evaluates the performance of nearly 1.8 million projections generated between February 2021 and November 2022. The results demonstrate that while unanticipated viral variants eventually invalidate scenario assumptions, the SMH provided a reliable 22-week window of guidance. Furthermore, multi-model ensembles consistently outperformed individual models, offering a robust framework for long-term public health planning.
Introduction
Throughout the COVID-19 pandemic, policymakers faced the challenge of making long-term decisions based on highly uncertain data. Traditional forecasting—predicting exactly what will happen—is generally only effective for 1–4 weeks. To address the need for longer-term strategic planning, the U.S. COVID-19 Scenario Modeling Hub shifted the focus to scenario modeling. By asking how the pandemic might unfold under specific conditions (e.g., high versus low vaccine booster uptake), the Hub provided a range of plausible futures that allowed for comparative evaluation of control measures.
Methodology
The SMH utilized a multi-model ensemble approach, drawing on the collective expertise of 4–9 independent modeling teams. These teams employed diverse methodologies, including mechanistic, agent-based, and statistical models.
Scenario Design
Each round of modeling typically utilized a 2x2 matrix of uncertainty. For example, scenarios might cross "High Variant Transmissibility" with "Low Vaccine Uptake" against "Low Variant Transmissibility" with "High Vaccine Uptake." This structure allowed researchers to isolate the impact of specific variables on the epidemic's trajectory.
The Trimmed Linear Opinion Pool (trimmed-LOP)
To aggregate these diverse projections, the Hub used a "trimmed Linear Opinion Pool." Unlike a simple arithmetic mean, this method averages the probability distributions provided by each team while "trimming" the most extreme high and low outliers. This process reduces variance and improves the "sharpness" of the ensemble's predictions, preventing a single idiosyncratic model from skewing the results.
Evaluation Metrics
Performance was assessed using the Weighted Interval Score (WIS), which balances accuracy with the appropriateness of the prediction interval width. The study also tracked Prediction Interval (PI) coverage—the frequency with which actual observed data fell within the models' predicted ranges.
Key Results and Performance
The Power of the Ensemble
The ensemble approach proved significantly more reliable than any single contributing model. The SMH ensemble achieved 79% coverage for 95% prediction intervals, whereas the median individual model coverage was only 40%. Individual models were frequently overconfident, providing intervals that were too narrow to capture the true variability of the pandemic. By averaging distributions, the ensemble naturally accounted for model-specific biases and structural uncertainties.
Comparative Accuracy
The ensemble outperformed a "naive" baseline model (which assumes current trends simply persist) by 39% to 58% across cases, hospitalizations, and deaths. Remarkably, the ensemble correctly identified the direction of the epidemic (increasing, flat, or decreasing) in 43% of weeks—a performance level comparable to short-term forecasts, despite the ensemble looking months further into the future.
The 22-Week Planning Horizon
A critical finding of the evaluation was the "validity window." On average, scenario assumptions remained valid for approximately 22 weeks before an unanticipated event—such as the emergence of the Delta or Omicron variants—fundamentally changed the epidemic's drivers. This suggests that scenario modeling can provide a reliable 5-month window for public health preparedness and policy formulation.
Operational Utility
Beyond statistical metrics, the SMH provided qualitative value to federal and state agencies. Even during periods of high uncertainty where quantitative accuracy was challenged (such as the initial Delta wave), the models correctly signaled impending resurgences. These projections were used operationally to guide bivalent booster recommendations and childhood vaccination policies, demonstrating that the value of a model lies as much in its ability to provide comparative risk assessments as in its raw predictive power.
Limitations and Future Directions
The primary constraint on the SMH was the "unknown unknown" of viral evolution. Unanticipated variants periodically reset the pandemic's trajectory, rendering previous scenario definitions obsolete. Additionally, while models were effective at predicting the scale of surges, they often struggled with the exact timing of wave peaks.
Future iterations of the Hub aim to:
- Increase Agility: Reduce the 5-week lag between scenario definition and the release of projections.
- Generalize Scenarios: Move away from specific variant names toward broader categories of "immune escape" to better capture potential viral diversity.
- Structural Refinement: Use retrospective "perfect hindsight" runs to determine whether errors stemmed from incorrect scenario assumptions or flaws in the underlying model structures.
Conclusion
The U.S. COVID-19 Scenario Modeling Hub represents a successful application of ensemble modeling to public health. By providing a structured way to navigate uncertainty, the Hub offered a critical 22-week window for policy planning. The success of the trimmed-LOP ensemble highlights the importance of collaborative, multi-model frameworks in responding to complex, evolving biological threats.
Summary updated