SIMLR: Machine Learning inside the SIR model for COVID-19 Forecasting

Roberto Vega, Leonardo Flores, Russell GreinerarXiv 2021 · OpenReview

TL;DR

This paper introduces SIMLR, a novel machine learning-enhanced SIR model that forecasts COVID-19 infections by dynamically incorporating government policy changes. SIMLR achieves state-of-the-art forecasting accuracy while remaining interpretable, offering a significant advancement for epidemic prediction and public health decision-making.

What Problem Does This Paper Address?

Accurate forecasting of infectious disease outbreaks, like COVID-19, is crucial for effective public health interventions and economic planning. Traditional epidemiological models, such as the SIR model, often struggle to capture the complex, dynamic nature of epidemics, especially when government policies (e.g., lockdowns, mask mandates) are frequently implemented and altered. These policy changes significantly influence disease transmission but are difficult to integrate into standard models. Existing machine learning approaches often treat policy data as static features or lack interpretability, making it hard to understand why a prediction is made. This paper aims to bridge this gap by developing a model that explicitly accounts for policy dynamics and provides interpretable forecasts.

Methods

The core of SIMLR is a probabilistic graphical model (PGM) that integrates a traditional SIR compartmental model with machine learning techniques. It operates on a "mixture of experts" principle:

  • Time-Varying SIR Parameters: Instead of using fixed transmission (β) and recovery (γ) rates, SIMLR estimates these parameters dynamically. This is achieved by transforming the SIR differential equations into difference equations and using a regression-based approach (Equation 5) to find optimal parameters for each time step, informed by historical data and optional regularization.
  • Policy-Informed Trend Prediction (CT): A key innovation is the "Change in Trend" (CT) variable, which predicts whether the epidemic's trajectory (increasing or decreasing infections) is likely to change. This prediction is based on past policy changes (CP) from 1 to 3 weeks prior.
  • Policy Change Forecasting (CP): SIMLR also forecasts the probability of future policy changes. This is crucial for longer-term predictions (3-4 weeks out) where current policy information is insufficient.
  • Urgency and Willingness (U & O): Auxiliary variables, "Urgency to change trend" (U) and "Openness to change" (O), are introduced to model the likelihood of policy shifts. U is modeled using a neural network conditional probability distribution (NN-CPD) that takes the number of new infections and their rate of change as input. O is modeled using conditional probability tables (CPTs) based on the duration since the last policy change.
  • Mixture of Experts: Based on the predicted CT value, SIMLR either uses the time-varying SIR model (if the trend is expected to continue) or a simpler "Same as Last Observed Week" (SLOW) model (if a trend change is anticipated). For longer forecasts, CT acts as a mixer, blending predictions from the time-varying SIR and SLOW models.

The model's interpretability stems from its PGM structure, allowing for the visualization and understanding of how policy changes influence trend predictions and, consequently, infection forecasts.

Datasets

The study utilized two primary sources of data:

  • OxCGRT (Oxford COVID-19 Government Response Tracker): This dataset provided information on government policies implemented in various regions, specifically focusing on "Workplace closing," "Stay at home requirements," and "Cancellation of public events." This data was used to train the policy-related components of the SIMLR model.
  • COVID-19 Data Repository (Johns Hopkins University): This repository supplied daily data on reported new infections and deaths, which was used for fitting the SIR model parameters and evaluating the forecasting performance.

The paper also mentions using publicly available predictions from the COVID-19 Forecast Hub for comparison.

Key Insights and Ideas

  • Policy Dynamics are Key: The most significant insight is that explicitly modeling government policy changes and their impact on epidemic trends is critical for accurate short-to-medium term forecasting. Simply using policy as static features is insufficient.
  • Interpretable ML is Achievable: SIMLR demonstrates that it's possible to build complex, high-performing forecasting models that are also interpretable, a crucial aspect for high-stakes decision-making in public health.
  • Forecasting Policy Changes is Feasible and Necessary: The ability to forecast the probability of future policy changes is essential for extending accurate predictions beyond a week or two.
  • SIR with Time-Varying Parameters is Sufficient: The authors argue that for COVID-19 forecasting, an SIR model with time-varying parameters, informed by policy, is adequate and doesn't necessarily require more complex compartmental models (like SEIR with additional compartments).
  • Simple Baselines are Hard to Beat: The study highlights that even simple forecasting methods like the "Same as Last Observed Week" (SLOW) can be surprisingly competitive, underscoring the difficulty of medium-term forecasting and the value of SIMLR's approach in combining strengths.

Key Results

  • Lower MAPE: SIMLR consistently achieved a lower Mean Average Percentage Error (MAPE) compared to the time-varying SIR (tf-v-SIR) and SLOW models for 1- to 4-week ahead forecasts across the United States and six Canadian provinces.
  • Competitive with State-of-the-Art: For US data, SIMLR's performance was comparable to the best models submitted to the CDC's COVID-19 Forecast Hub, with no statistically significant difference in MAPE compared to the top-performing LNQ-ens1 model.
  • Effective Policy Integration: The model demonstrated its ability to adapt to policy changes, correctly predicting trend shifts and improving accuracy, particularly in regions with dynamic policy interventions. For instance, in Alberta, SIMLR correctly predicted a change in trend around weeks 33-35 after a policy relaxation.
  • Interpretability: The PGM structure of SIMLR allows for clear visualization of how policy changes influence trend predictions, making the model's reasoning transparent.

Limitations and Future Work

  • Data Scarcity for CPTs: A significant limitation is the scarcity of historical data for learning the conditional probability tables (CPTs) for discrete variables like CT, CP, and O. This forced the authors to rely on domain knowledge and probabilistic labels, which could lead to inaccuracies if these estimations are wrong.
  • Regional Thresholds: The "thresholds" for government action might differ across regions, and the CPTs were set a priori for US and Canada. These might need to be adjusted for other countries.
  • Causality vs. Correlation: While SIMLR models the influence of policy on trends, the PGM structure does not imply causality. The authors acknowledge that the relationship between policy and trends is likely bidirectional.
  • Identifying Specific Policies: SIMLR can estimate the likelihood of a policy change but does not identify which specific policy might be implemented next.

Future work could involve developing more robust methods for learning CPTs with limited data, exploring regional variations in policy thresholds, and potentially extending the model to predict specific policy actions. The authors also suggest that this approach can be generalized to forecast other infectious diseases.

Summary updated

Abstract

Accurate forecasts of the number of newly infected people during an epidemic are critical for making effective timely decisions. This paper addresses this challenge using the SIMLR model, which incorporates machine learning (ML) into the epidemiological SIR model. For each region, SIMLR tracks the changes in the policies implemented at the government level, which it uses to estimate the time-varying parameters of an SIR model for forecasting the number of new infections 1- to 4-weeks in advance.It also forecasts the probability of changes in those government policies at each of these future times, which is essential for the longer-range forecasts. We applied SIMLR to data from regions in Canada and in the United States,and show that its MAPE (mean average percentage error) performance is as good as SOTA forecasting models, with the added advantage of being an interpretable model. We expect that this approach will be useful not only for forecasting COVID-19 infections, but also in predicting the evolution of other infectious diseases.
BibTeX
@misc{vega2021simlr,
  author = {Roberto Vega and Leonardo Flores and Russell Greiner},
  title = {{SIMLR: Machine Learning inside the SIR model for COVID-19 Forecasting}},
  year = {2021},
  eprint = {2106.01590},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.01590},
}
Figure