Ensemble forecasts of COVID-19 activity to support Australia’s pandemic response: 2020–22

Robert Moss, Ruarai Tobin, Mitchell O’Hara-Wild, Adeshina I. Adekunle, Dennis Liu, Tobin South, Dylan Morris, Gerard E. Ryan, Tianxiao Hao, Addanki Jyothi Babu, Katharine L. Senior, James Wood, Nick Golding, Joshua V. Ross, Peter Dawson, Rob J. Hyndman, David J. Price, James M. McCaw, Freya M. ShearermedRxiv 2025 · DOI

TL;DR

During the COVID-19 pandemic, Australia relied on a collaborative consortium of researchers to produce weekly ensemble forecasts of case counts. By combining diverse modeling approaches—from mechanistic SEIR models to statistical autoregression—the ensemble consistently outperformed individual models, providing a robust and interpretable evidence base for national and jurisdictional public health decisions.

What Problem Does This Paper Address?

Predicting the trajectory of an infectious disease in real-time is notoriously difficult. Individual models often carry inherent biases or struggle with "change points" (sudden shifts in transmission). For policymakers, relying on a single model is risky.

This paper evaluates the performance of Australia’s "consortium" approach: a multi-model ensemble used from 2020 to 2022. The authors address how to combine disparate models effectively, how to maintain forecast quality through a human-in-the-loop peer-review process, and how to communicate complex probabilistic outputs to government stakeholders who must make high-stakes decisions like easing lockdowns or preparing hospital capacity.

Methods

The researchers utilized an equal-weight ensemble approach. Each week, four independent modeling teams submitted 2,000 simulated trajectories for a 28-day forecast horizon across Australia’s eight states and territories.

  • The Models: The ensemble combined two stochastic compartmental (SEIR-type) models (from DST and UoM), a branching process model (UoA), and a global autoregressive model (MU).
  • The Process: Unlike "black box" automated ensembles, this featured a weekly internal peer-review process. The consortium met to discuss model performance, data quality issues, and whether specific model iterations should be excluded if they were inappropriately sensitive to reporting delays or produced suspect predictions.
  • Evaluation Metrics: Forecasts were assessed using Continuous Ranked Probability Scores (CRPS) to measure accuracy, along with measures of bias (tendency to over/undershoot) and coverage (how often the true value fell within predicted credible intervals).

Datasets

The primary data source was the National Notifiable Disease Surveillance System (NNDSS), which provided de-identified line lists of reported cases.

  • Key variables: Jurisdiction, date of symptom onset (imputed when missing), date of notification, and whether the infection was acquired locally or overseas.
  • Data Quality Shift: The paper notes a significant challenge as the pandemic progressed: symptom onset reporting dropped from 82% of cases in 2020 to just 38% in 2022 as case volumes surged and "reopening" began.

Key Insights and Ideas

  • The "Wisdom of the Crowd" (with a Filter): The ensemble was rarely the single "best" model for any given day, but it was almost always the most reliable. It was ranked in the top three models 98.7% of the time.
  • Human-in-the-Loop is Essential: The peer-review process allowed the team to explain why a forecast looked a certain way (e.g., "Model X is reacting to a reporting lag in Victoria"). This built trust with public health stakeholders that a purely algorithmic approach might have lacked.
  • The "Peak" Problem: All models struggled to predict the exact timing and height of epidemic peaks. Forecasts tended to overshoot peaks and undershoot the initial onset, particularly during the volatile Omicron BA.1 wave.
  • Visual Communication as a Tool: The authors found that standard "fan charts" (credible intervals) could be misinterpreted as single trajectories. They refined their reporting to include density plots for peak timing and overlaid individual sample trajectories to better convey uncertainty.

Key Results

  • Superior Reliability: The ensemble outperformed individual models consistently across different waves (Pre-Delta, Delta, and Omicron).
  • Ranking: While individual models were more likely to be the #1 ranked forecast for a specific day, the ensemble was at least 2nd-best approximately 66.7% of the time.
  • Lead Time Accuracy: As expected, accuracy improved as the lead time shortened. The mean CRPS was 0.501 at 4 weeks out, improving to 0.184 at 1 week out.
  • Skill Scores: Mechanistic models performed exceptionally well during periods of zero cases (median skill scores of 0.75–0.97), but the statistical autoregressive model (MU) often provided better median performance during active transmission.

Limitations and Future Work

  • Predicting Change Points: The primary limitation remains the inability to accurately predict "inflection points"—the exact moment an epidemic starts to turn.
  • Waning Data Quality: As the response shifted toward "living with COVID," the granularity of surveillance data (like symptom onset) decreased, making mechanistic modeling more difficult.
  • Immune Complexity: By 2022, the "immune landscape" became incredibly complex due to varying vaccination levels and prior infections, making it harder to estimate population susceptibility.
  • Future Direction: The authors suggest that while this small-consortium model worked well for Australia, scaling such a "human-heavy" peer-review process to larger regions (like the US or Europe) remains a significant challenge.

Summary updated

Abstract

Abstract During the COVID-19 pandemic, many countries used real-time data analyses, predictive modelling, and COVID-19 case forecasts, to incorporate emerging evi-dence into their decisions. In Australia, national and jurisdictional public health re-sponses were informed by weekly ensemble forecasts of daily COVID-19 case counts for each of Australia’s eight states and territories, produced by a consortium of researchers under contract with the Australian Government. We evaluated ap-proximately 100,000 predictions for daily case counts 1–28 days into the future, generated between July 2020 and December 2022, and report here (a) how the ensemble forecasts supported public health responses; (b) how well the ensemble forecast performed, relative to the forecasts produced by each contributing team; and (c) how we refined our reporting and visualisations to ensure that outputs were interpreted appropriately. Similar to COVID-19 forecasting studies in other coun-tries, we found that the ensemble forecast consistently out-performed the individual model forecasts, and that performance was lowest when there were rapid changes in the epidemiology, such as periods around epidemic peaks. Our consortium’s inter-nal peer-review process allowed us to explain how features of each ensemble forecast related to the design of the individual models, and this helped enable public health stakeholders to interpret the forecasts appropriately. Ultimately, our forecasts pro-vided information that supported public health responses during periods of different policy goals, and over a wide range of epidemic scenarios.
BibTeX
@misc{moss2025ensemble,
  author = {Robert Moss and Ruarai Tobin and Mitchell O’Hara-Wild and Adeshina I. Adekunle and Dennis Liu and Tobin South and Dylan Morris and Gerard E. Ryan and Tianxiao Hao and Addanki Jyothi Babu and Katharine L. Senior and James Wood and Nick Golding and Joshua V. Ross and Peter Dawson and Rob J. Hyndman and David J. Price and James M. McCaw and Freya M. Shearer},
  title = {{Ensemble forecasts of COVID-19 activity to support Australia’s pandemic response: 2020–22}},
  year = {2025},
  doi = {10.1101/2025.09.10.25335544},
  howpublished = {medRxiv},
}
Figure