Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
Katharine Sherratt, Hugo Gruson, Rok Grah, Hillary Johnson, Rene Niehus, Bastian Prasse, F. Sandman, Jannik Deuschel, Daniel Wolffram, Sam Abbott, Alexander Ullrich, Graham Gibson, El. Ray, Ng. Reich, Daniel Sheldon, Yijin Wang, Nutcha Wattanachit, L. Wang, Ján Trnka, Guillaume Obozinski, Tao Sun, Dorina Thanou, Laurence Pottier, Ekaterina Krymova, Maria Vittoria Barbarossa, Neele Leithäuser, Jan Möhring, Johanna Schneider, Jarosław Wlazło, Jan Fuhrmann, Berit Lange, Isti Rodiah, Prasith Baccam, Heidi Gurung, Steven A. Stage, Brad T. Suchoski, Jozef Budzinski, Robert Walraven, Inmaculada Villanueva, Vít Tuček, Martin Šmíd, Milan Zajíček, C. Pérez Álvarez, Borja Reina, Nikos I Bosse, Sophie Meakin, Pierfrancesco Alaimo Di Loro, Antonello Maruotti, Veronika Eclerová, Andrea Kraus, David Kraus, Lenka Přibylová, Babalis Dimitris, Ml. Li, Soni Saksham, Jonas Dehning, Sebastian Mohr, Viola Priesemann, Grzegorz Redlarski, Benjamı́n Béjar, Giovanni Ardenghi, Nicola Parolini, Giovanni Ziarelli, Wolfgang Böck, Stefan Heyder, Thomas Hotz, David E Singh, Miguel Guzmán-Merino, Jose L Aznarte, David Moriña, Sergio Alonso, E. Álvarez, Daniel López, Clara Prats, Jp. Burgard, Arne Rodloff, Thomas Zimmermann, Alexander Kuhlmann, Janez Žibert, Fulvia Pennoni, Fabio Divino, Martí Català, Gianfranco Lovison, Paolo Giudici, Barbara Tarantino, Francesco Bartolucci, Giovanna Jona Lasinio, Marco Mingione, Alessio Farcomeni, Ajitesh Srivastava, Pablo Montero‐Manso, Aniruddha Adiga, Benjamin Hurt, Bryan Lewis, Madhav Marathe, Przemyslaw Porebski, Srinivasan Venkatramanan, Rafał Bartczuk, Filip Dreger, Anna GambinmedRxiv2022 · DOI
Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. We used open-source tools to develop a public European COVID-19 Forecast Hub, inviting groups globally to contribute weekly forecasts for COVID-19 cases and deaths. Over 52 weeks, we collected and combined up to 28 forecast models for 32 countries. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative Weighted Interval Score (WIS) than 84% of participating models’ forecasts of incident cases and 92% of participating models’ forecasts of deaths. Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics.
Introduction
Predicting the trajectory of an infectious disease is notoriously difficult. During the COVID-19 pandemic, hundreds of independent research teams developed models using different assumptions, data sources, and mathematical frameworks. For policymakers, this variety of forecasts can be paralyzing: which model should be trusted when they disagree? This study evaluates the European COVID-19 Forecast Hub, a centralized infrastructure designed to standardize these predictions and combine them into a "consensus" ensemble, testing if the "wisdom of the crowd" provides more stable situational awareness than individual models.
Methods
The researchers established a standardized pipeline where teams submitted weekly forecasts for incident cases and deaths for the upcoming 1–4 weeks. Teams submitted 23 specific quantiles (from 0.01 to 0.99) to represent their uncertainty. The Hub created a weekly ensemble by calculating the average of these quantiles. Initially, an arithmetic mean was used, but the methodology shifted to the median in July 2021 to prevent outlier models from skewing results. Performance was measured using the Weighted Interval Score (WIS), which penalizes both inaccuracy and poor calibration (intervals being too wide or too narrow). A relative WIS was used to allow for fair ranking across models with different coverage.
Results
The study analyzed forecasts for 32 countries over a 52-week period covering the emergence of the Delta and Omicron variants. The ensemble consistently outperformed the vast majority of individual models and a "naive" baseline.
Figure 2: Performance metrics (WIS and coverage) of the Hub ensemble compared to individual models across 1-4 week forecast horizons.
As shown in Figure 2, the ensemble (purple) maintained lower relative WIS and better calibration than the aggregate of individual models (blue). A key finding was that death forecasts remained stable up to four weeks ahead, whereas case forecast accuracy dropped sharply after two weeks. This is likely because deaths are a lagging indicator; the infections leading to deaths in week four have often already occurred by the time the forecast is made.
Geographic Consistency
The ensemble's dominance was not limited to specific regions. Across 32 countries, the ensemble frequently outperformed all of its individual component models.
Figure 3: Scaled relative WIS across 32 European countries. Black stars represent the ensemble performance, consistently appearing below the baseline (1.0) and most individual model distributions.
The study highlights that using the median to combine models is superior to the mean in epidemiology, as it remains robust against "wild" results from individual models. Interestingly, complex weighted ensembles based on past performance did not consistently beat the simple, unweighted median ensemble, suggesting that past performance is not a perfect predictor of future accuracy in rapidly changing epidemic landscapes.
While the ensemble provides a powerful tool for public health, limitations remain, including frequent revisions to ground-truth data and the "observer effect," where public forecasts may lead to behavioral changes that invalidate the original prediction. Future work will focus on expanding the Hub to include hospitalization forecasts and better integrating vaccination rates into the ensemble logic.
Summary updated
Abstract
Abstract Background Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported from a standardised source over the next one to four weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance. Results Over 52 weeks we collected and combined up to 28 forecast models for 32 countries. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 84% of participating models’ forecasts of incident cases (with a total N=862), and 92% of participating models’ forecasts of deaths (N=746). Across a one to four week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over four weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than two weeks. Code and data availability All data and code are publicly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.
BibTeX
@misc{sherratt2022predictive,
author = {Katharine Sherratt and Hugo Gruson and Rok Grah and Hillary Johnson and Rene Niehus and Bastian Prasse and F. Sandman and Jannik Deuschel and Daniel Wolffram and Sam Abbott and Alexander Ullrich and Graham Gibson and EL. Ray and NG. Reich and Daniel Sheldon and Yijin Wang and Nutcha Wattanachit and L. Wang and Ján Trnka and Guillaume Obozinski and Tao Sun and Dorina Thanou and Laurence Pottier and Ekaterina Krymova and Maria Vittoria Barbarossa and Neele Leithäuser and Jan Möhring and Johanna Schneider and Jarosław Wlazło and Jan Fuhrmann and Berit Lange and Isti Rodiah and Prasith Baccam and Heidi Gurung and Steven A. Stage and Brad T. Suchoski and Jozef Budzinski and Robert Walraven and Inmaculada Villanueva and Vít Tuček and Martin Šmíd and Milan Zajíček and C. Pérez Álvarez and Borja Reina and Nikos I Bosse and Sophie Meakin and Pierfrancesco Alaimo Di Loro and Antonello Maruotti and Veronika Eclerová and Andrea Kraus and David Kraus and Lenka Přibylová and Babalis Dimitris and ML. Li and Soni Saksham and Jonas Dehning and Sebastian Mohr and Viola Priesemann and Grzegorz Redlarski and Benjamı́n Béjar and Giovanni Ardenghi and Nicola Parolini and Giovanni Ziarelli and Wolfgang Böck and Stefan Heyder and Thomas Hotz and David E Singh and Miguel Guzmán-Merino and Jose L Aznarte and David Moriña and Sergio Alonso and E. Álvarez and Daniel López and Clara Prats and JP. Burgard and Arne Rodloff and Thomas Zimmermann and Alexander Kuhlmann and Janez Žibert and Fulvia Pennoni and Fabio Divino and Martí Català and Gianfranco Lovison and Paolo Giudici and Barbara Tarantino and Francesco Bartolucci and Giovanna Jona Lasinio and Marco Mingione and Alessio Farcomeni and Ajitesh Srivastava and Pablo Montero‐Manso and Aniruddha Adiga and Benjamin Hurt and Bryan Lewis and Madhav Marathe and Przemyslaw Porebski and Srinivasan Venkatramanan and Rafał Bartczuk and Filip Dreger and Anna Gambin},
title = {{Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations}},
year = {2022},
doi = {10.1101/2022.06.16.22276024},
howpublished = {medRxiv},
}