Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub

Emily Howerton, Lucie Contamin, Luke C. Mullany, Michelle Qin, Nicholas G Reich, Samantha J. Bents, Rebecca K. Borchering, Sung-mok Jung, Sara L. Loo, Claire P. Smith, John Levander, Jessica Kerr, Jeremy U. Espino, Willem G. van Panhuis, Harry Hochheiser, Marta Galanti, Teresa K. Yamana, Sen Pei, Jeffrey Shaman, Kaitlin Rainwater‐Lovett, Matt Kinsey, Kate Tallaksen, Shelby Wilson, Lauren Shin, Joseph C. Lemaitre, Joshua Kaminsky, Juan Dent, Elizabeth C. Lee, Clif D. McKee, Alison L. Hill, D. Karlen, Matteo Chinazzi, Jessica T. Davis, Kunpeng Mu, Xinyue Xiong, Ana Pastore y Piontti, Alessandro Vespignani, Erik Rosenstrom, Julie S. Ivy, María E. Mayorga, Julie Swann, Guido España, Sean Cavany, Sean M. Moore, T. Alex Perkins, Thomas J. Hladish, Alexander N. Pillai, Kok Ben Toh, Ira M. Longini, Shi Chen, Rajib Paul, Daniel Janies, Jean‐Claude Thill, Anass Bouchnita, Kaiming Bi, Michael Lachmann, Spencer J. Fox, Lauren Ancel Meyers, Ajitesh Srivastava, Przemyslaw Porebski, Srini Venkatramanan, Aniruddha Adiga, Bryan Lewis, Brian Klahn, Joseph Outten, Benjamin Hurt, Jiangzhuo Chen, Henning Mortveit, Amanda Wilson, Madhav Marathe, Stefan Hoops, Parantapa Bhattacharya, Dustin Machi, Betsy L. Cadwell, Jessica M. Healy, Rachel B. Slayton, Michael A. Johansson, Matthew Biggerstaff, Shaun Truelove, Michael C. Runge, Katriona Shea, Cécile Viboud, Justin LesslermedRxiv 2023 · DOI

Abstract

Predicting the trajectory of an epidemic beyond a few weeks is inherently limited by the complexity of disease systems and the unpredictability of human behavior. While short-term forecasts rely on current momentum, long-term projections require a different approach. The U.S. COVID-19 Scenario Modeling Hub (SMH) was established to provide 6-month-ahead projections by defining specific "what-if" scenarios regarding vaccine uptake and variant transmissibility. This paper evaluates the performance of nearly 1.8 million projections generated between February 2021 and November 2022. The results demonstrate that while unanticipated viral variants eventually invalidate scenario assumptions, the SMH provided a reliable 22-week window of guidance. Furthermore, multi-model ensembles consistently outperformed individual models, offering a robust framework for long-term public health planning.

Introduction

Throughout the COVID-19 pandemic, policymakers faced the challenge of making long-term decisions based on highly uncertain data. Traditional forecasting—predicting exactly what will happen—is generally only effective for 1–4 weeks. To address the need for longer-term strategic planning, the U.S. COVID-19 Scenario Modeling Hub shifted the focus to scenario modeling. By asking how the pandemic might unfold under specific conditions (e.g., high versus low vaccine booster uptake), the Hub provided a range of plausible futures that allowed for comparative evaluation of control measures.

Methodology

The SMH utilized a multi-model ensemble approach, drawing on the collective expertise of 4–9 independent modeling teams. These teams employed diverse methodologies, including mechanistic, agent-based, and statistical models.

Scenario Design

Each round of modeling typically utilized a 2x2 matrix of uncertainty. For example, scenarios might cross "High Variant Transmissibility" with "Low Vaccine Uptake" against "Low Variant Transmissibility" with "High Vaccine Uptake." This structure allowed researchers to isolate the impact of specific variables on the epidemic's trajectory.

The Trimmed Linear Opinion Pool (trimmed-LOP)

To aggregate these diverse projections, the Hub used a "trimmed Linear Opinion Pool." Unlike a simple arithmetic mean, this method averages the probability distributions provided by each team while "trimming" the most extreme high and low outliers. This process reduces variance and improves the "sharpness" of the ensemble's predictions, preventing a single idiosyncratic model from skewing the results.

Evaluation Metrics

Performance was assessed using the Weighted Interval Score (WIS), which balances accuracy with the appropriateness of the prediction interval width. The study also tracked Prediction Interval (PI) coverage—the frequency with which actual observed data fell within the models' predicted ranges.

Key Results and Performance

The Power of the Ensemble

The ensemble approach proved significantly more reliable than any single contributing model. The SMH ensemble achieved 79% coverage for 95% prediction intervals, whereas the median individual model coverage was only 40%. Individual models were frequently overconfident, providing intervals that were too narrow to capture the true variability of the pandemic. By averaging distributions, the ensemble naturally accounted for model-specific biases and structural uncertainties.

Comparative Accuracy

The ensemble outperformed a "naive" baseline model (which assumes current trends simply persist) by 39% to 58% across cases, hospitalizations, and deaths. Remarkably, the ensemble correctly identified the direction of the epidemic (increasing, flat, or decreasing) in 43% of weeks—a performance level comparable to short-term forecasts, despite the ensemble looking months further into the future.

The 22-Week Planning Horizon

A critical finding of the evaluation was the "validity window." On average, scenario assumptions remained valid for approximately 22 weeks before an unanticipated event—such as the emergence of the Delta or Omicron variants—fundamentally changed the epidemic's drivers. This suggests that scenario modeling can provide a reliable 5-month window for public health preparedness and policy formulation.

Operational Utility

Beyond statistical metrics, the SMH provided qualitative value to federal and state agencies. Even during periods of high uncertainty where quantitative accuracy was challenged (such as the initial Delta wave), the models correctly signaled impending resurgences. These projections were used operationally to guide bivalent booster recommendations and childhood vaccination policies, demonstrating that the value of a model lies as much in its ability to provide comparative risk assessments as in its raw predictive power.

Limitations and Future Directions

The primary constraint on the SMH was the "unknown unknown" of viral evolution. Unanticipated variants periodically reset the pandemic's trajectory, rendering previous scenario definitions obsolete. Additionally, while models were effective at predicting the scale of surges, they often struggled with the exact timing of wave peaks.

Future iterations of the Hub aim to:

  • Increase Agility: Reduce the 5-week lag between scenario definition and the release of projections.
  • Generalize Scenarios: Move away from specific variant names toward broader categories of "immune escape" to better capture potential viral diversity.
  • Structural Refinement: Use retrospective "perfect hindsight" runs to determine whether errors stemmed from incorrect scenario assumptions or flaws in the underlying model structures.

Conclusion

The U.S. COVID-19 Scenario Modeling Hub represents a successful application of ensemble modeling to public health. By providing a structured way to navigate uncertainty, the Hub offered a critical 22-week window for policy planning. The success of the trimmed-LOP ensemble highlights the importance of collaborative, multi-model frameworks in responding to complex, evolving biological threats.

Summary updated

Abstract

Abstract Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.
BibTeX
@misc{howerton2023informing,
  author = {Emily Howerton and Lucie Contamin and Luke C. Mullany and Michelle Qin and Nicholas G Reich and Samantha J. Bents and Rebecca K. Borchering and Sung-mok Jung and Sara L. Loo and Claire P. Smith and John Levander and Jessica Kerr and Jeremy U. Espino and Willem G. van Panhuis and Harry Hochheiser and Marta Galanti and Teresa K. Yamana and Sen Pei and Jeffrey Shaman and Kaitlin Rainwater‐Lovett and Matt Kinsey and Kate Tallaksen and Shelby Wilson and Lauren Shin and Joseph C. Lemaitre and Joshua Kaminsky and Juan Dent and Elizabeth C. Lee and Clif D. McKee and Alison L. Hill and D. Karlen and Matteo Chinazzi and Jessica T. Davis and Kunpeng Mu and Xinyue Xiong and Ana Pastore y Piontti and Alessandro Vespignani and Erik Rosenstrom and Julie S. Ivy and María E. Mayorga and Julie Swann and Guido España and Sean Cavany and Sean M. Moore and T. Alex Perkins and Thomas J. Hladish and Alexander N. Pillai and Kok Ben Toh and Ira M. Longini and Shi Chen and Rajib Paul and Daniel Janies and Jean‐Claude Thill and Anass Bouchnita and Kaiming Bi and Michael Lachmann and Spencer J. Fox and Lauren Ancel Meyers and Ajitesh Srivastava and Przemyslaw Porebski and Srini Venkatramanan and Aniruddha Adiga and Bryan Lewis and Brian Klahn and Joseph Outten and Benjamin Hurt and Jiangzhuo Chen and Henning Mortveit and Amanda Wilson and Madhav Marathe and Stefan Hoops and Parantapa Bhattacharya and Dustin Machi and Betsy L. Cadwell and Jessica M. Healy and Rachel B. Slayton and Michael A. Johansson and Matthew Biggerstaff and Shaun Truelove and Michael C. Runge and Katriona Shea and Cécile Viboud and Justin Lessler},
  title = {{Informing pandemic response in the face of uncertainty. <i>An evaluation of the U.S. COVID-19 Scenario Modeling Hub</i>}},
  year = {2023},
  doi = {10.1101/2023.06.28.23291998},
  howpublished = {medRxiv},
}
Figure