Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Emily Howerton, Lucie Contamin, Luke C. Mullany, Michelle Qin, Nicholas G Reich, Samantha J. Bents, Rebecca K. Borchering, Sung-mok Jung, Sara L. Loo, Claire P. Smith, John Levander, Jessica Kerr, Jeremy U. Espino, Willem G. van Panhuis, Harry Hochheiser, Marta Galanti, Teresa K. Yamana, Sen Pei, Jeffrey Shaman, Kaitlin Rainwater‐Lovett, Matt Kinsey, Kate Tallaksen, Shelby Wilson, Lauren Shin, Joseph C. Lemaitre, Joshua Kaminsky, Juan Dent, Elizabeth C. Lee, Clifton McKee, Alison L. Hill, D. Karlen, Matteo Chinazzi, Jessica T. Davis, Kunpeng Mu, Xinyue Xiong, Ana Pastore y Piontti, Alessandro Vespignani, Erik Rosenstrom, Julie S. Ivy, María E. Mayorga, Julie Swann, Guido España, Sean Cavany, Sean M. Moore, T. Alex Perkins, Thomas J. Hladish, Alexander N. Pillai, Kok Ben Toh, Ira M. Longini, Shi Chen, Rajib Paul, Daniel Janies, Jean‐Claude Thill, Anass Bouchnita, Kaiming Bi, Michael Lachmann, Spencer J. Fox, Lauren Ancel Meyers, Ajitesh Srivastava, Przemyslaw Porebski, Srini Venkatramanan, Aniruddha Adiga, Bryan Lewis, Brian Klahn, Joseph Outten, Benjamin Hurt, Jiangzhuo Chen, Henning Mortveit, Amanda Wilson, Madhav Marathe, Stefan Hoops, Parantapa Bhattacharya, Dustin Machi, Betsy L. Cadwell, Jessica M. Healy, Rachel B. Slayton, Michael A. Johansson, Matthew Biggerstaff, Shaun Truelove, Michael C. Runge, Katriona Shea, Cécile Viboud, Justin LesslerNature Communications 2023 · DOI

Abstract: Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Introduction

Predicting the trajectory of an epidemic is notoriously difficult because it depends on a "moving target" of human behavior, policy changes, and viral evolution. Standard forecasting models usually fail beyond a 4-week horizon because they cannot account for these shifting drivers. To address this, the US COVID-19 Scenario Modeling Hub (SMH) utilized scenario modeling. Instead of a single "best guess," modeling teams projected outcomes based on specific "if-then" conditions (e.g., varying levels of vaccine uptake or variant transmissibility). This study provides a comprehensive retrospective on 16 rounds of SMH projections, evaluating how well these models performed when their underlying assumptions matched reality.

Methods

The SMH operated as a collaborative hub, convening 4 to 9 independent modeling teams per round. To synthesize these diverse inputs, the researchers used a Linear Opinion Pool (LOP) ensemble method. Specifically, they employed a "trimmed-LOP" that excludes extreme outliers while preserving the variation between different models to better represent true uncertainty.

The SMH process: from prospective scenario design to retrospective evaluation of plausibility and divergence.

Evaluation required disentangling scenario validity (did the assumptions match reality?) from model calibration (given the assumptions, was the math right?). The authors identified "plausible scenario-weeks"—periods where the scenario assumptions, such as vaccine rollout speed, actually aligned with real-world data. Performance was measured using the Weighted Interval Score (WIS) and Prediction Interval (PI) coverage.

Key Results

The analysis revealed that scenarios typically remained "close to reality" for an average of 22 weeks. Beyond this "variant horizon," the emergence of unanticipated variants like Alpha, Delta, or Omicron generally invalidated the underlying scenario assumptions.

US incident hospitalizations and SMH projection rounds across major variant waves.

The Wisdom of the Ensemble

The multi-model ensemble was consistently more reliable than any individual model. While individual models were often "overconfident" (with a median 95% PI coverage of only 40-42%), the ensemble’s uncertainty intervals were much closer to the ideal target, covering the truth approximately 80% of the time during plausible weeks. Furthermore, the SMH ensemble outperformed naive baseline models by 39% to 58% across cases, hospitalizations, and deaths.

Strategic Bracketing

The hub was most successful when it "bracketed" reality—designing scenarios such that one was more pessimistic and another more optimistic than the eventual reality. This ensured the actual trajectory fell within the range of projected possibilities. Interestingly, in 43% of cases, the ensemble performed well even when scenario assumptions were incorrect, suggesting that model and assumption errors occasionally cancelled each other out.

Discussion and Limitations

While the SMH provided vital guidance for pandemic policy, the study highlights several challenges. A lack of high-quality, real-time data on Non-Pharmaceutical Interventions (NPIs) like mask mandates made it difficult to evaluate policy-specific scenarios. Additionally, the retrospective nature of identifying "plausible" weeks could potentially favor scenario models over traditional forecasts.

The authors conclude that "peace time" research and infrastructure are essential. Establishing robust systems to collect behavioral and viral data before the next pandemic is critical to ensuring that scenario assumptions can be rapidly updated as new threats emerge.

Summary updated

Abstract

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.
BibTeX
@inproceedings{howerton2023evaluation,
  author = {Emily Howerton and Lucie Contamin and Luke C. Mullany and Michelle Qin and Nicholas G Reich and Samantha J. Bents and Rebecca K. Borchering and Sung-mok Jung and Sara L. Loo and Claire P. Smith and John Levander and Jessica Kerr and Jeremy U. Espino and Willem G. van Panhuis and Harry Hochheiser and Marta Galanti and Teresa K. Yamana and Sen Pei and Jeffrey Shaman and Kaitlin Rainwater‐Lovett and Matt Kinsey and Kate Tallaksen and Shelby Wilson and Lauren Shin and Joseph C. Lemaitre and Joshua Kaminsky and Juan Dent and Elizabeth C. Lee and Clifton McKee and Alison L. Hill and D. Karlen and Matteo Chinazzi and Jessica T. Davis and Kunpeng Mu and Xinyue Xiong and Ana Pastore y Piontti and Alessandro Vespignani and Erik Rosenstrom and Julie S. Ivy and María E. Mayorga and Julie Swann and Guido España and Sean Cavany and Sean M. Moore and T. Alex Perkins and Thomas J. Hladish and Alexander N. Pillai and Kok Ben Toh and Ira M. Longini and Shi Chen and Rajib Paul and Daniel Janies and Jean‐Claude Thill and Anass Bouchnita and Kaiming Bi and Michael Lachmann and Spencer J. Fox and Lauren Ancel Meyers and Ajitesh Srivastava and Przemyslaw Porebski and Srini Venkatramanan and Aniruddha Adiga and Bryan Lewis and Brian Klahn and Joseph Outten and Benjamin Hurt and Jiangzhuo Chen and Henning Mortveit and Amanda Wilson and Madhav Marathe and Stefan Hoops and Parantapa Bhattacharya and Dustin Machi and Betsy L. Cadwell and Jessica M. Healy and Rachel B. Slayton and Michael A. Johansson and Matthew Biggerstaff and Shaun Truelove and Michael C. Runge and Katriona Shea and Cécile Viboud and Justin Lessler},
  title = {{Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-42680-x},
}
Figure