Markovian And Non-Markovian Processes with Active Decision Making Strategies For Addressing The COVID-19 Pandemic

Hamid Eftekhari, Debarghya Mukherjee, Moulinath Banerjee, Ya’acov RitovarXiv 2020 · DOI · arXiv

Abstract

We study and predict the evolution of Covid-19 in six US states from the period May 1 through August 31 using a discrete compartment-based model and prescribe active intervention policies, like lockdowns, on the basis of minimizing a loss function, within the broad framework of partially observed Markov decision processes. For each state, Covid-19 data for 40 days are analyzed to estimate the transition probabilities between compartments and other parameters associated with the evolution of the epidemic. These quantities are then used to predict the course of the epidemic in the given state for the next 50 days under various policy allocations. Our analysis shows that none of the six states need lockdowns over the test period, though the no lockdown prescription is to be interpreted with caution: responsible mask use and social distancing of course need to be continued. A sketch of a non-Markovian formulation of Covid-19 propagation is presented as an attractive avenue for future research.

Introduction

During the height of the COVID-19 pandemic, governments faced a grueling dilemma: how to curb the virus without destroying the economy. Traditional epidemiological models (like SIR) are excellent at predicting spread but often lack an "active" decision-making component. Furthermore, these models often assume we know the exact state of the population. In reality, we only see a fraction of the truth through testing data—a "partially observed" reality. This paper addresses the need for a mathematical framework that can ingest noisy testing data, estimate the hidden "true" number of infections, and output an optimal policy (lockdown vs. no lockdown) based on a specific loss function.

Methods

The authors employ two distinct modeling approaches to address the pandemic's dynamics.

The Markovian Model (POMDP)

This is a discrete-time compartmental model that extends the SEIRD structure. Crucially, the researchers split the "Infected" compartment into mildly infected ($I_m$) and severely ill ($I_s$). This distinction is vital for modeling healthcare system strain and hospital capacity.

Flow diagram of the compartmental model showing transitions between Susceptible (S), Latent (L), Mildly Infected (Im), Severely Infected (Is), Recovered (R), and Deceased (D) states.

The model utilizes "bang-bang" control policies, where lockdowns are triggered or lifted based on whether a weighted combination of mild and severe cases crosses specific thresholds. The model minimizes a loss function that sums the economic cost of lockdowns, the statistical cost of life, and the "cost" of treatment denial (which spikes if hospital capacity is exceeded).

The Non-Markovian Sketch

Recognizing that virus transitions (like incubation periods) do not follow simple "memoryless" probabilities, the authors propose a more complex model that tracks individuals separately. This allows for spatial heterogeneity (modeling clusters like counties) and realistic time delays for disease progression.

Datasets and Experimental Setup

The study utilizes real-world COVID-19 data from the Johns Hopkins University Center for Systems Science and Engineering (CSSE) repository for six US states: Michigan (MI), New Jersey (NJ), Florida (FL), Arizona (AZ), Texas (TX), and California (CA). The training period involved 40 days of data (starting May or June 2020), followed by a 50-day test period. Economic costs were scaled based on state GDP, using a baseline estimate of $7 trillion/year for a full US lockdown.

Key Results

  • Prescribed Policy: For the period studied (Summer 2020), the model suggested that none of the six states required full lockdowns. The economic cost of shutting down was found to be higher than the projected health costs, provided that hospital beds remained available.
  • The Nuance of "No Lockdown": The authors emphasize that a "no lockdown" prescription does not mean a return to pre-pandemic life. It assumes an $R_0$ of around 1.8—lower than the initial 2.5+—because it assumes the public continues "responsible behavior" like masking and social distancing.
  • Sensitivity to Parameters: A major finding is how sensitive the optimal policy is to the proportion of severe cases ($p_s$). In a perturbation analysis for Michigan, increasing the assumed hospitalization rate from 7% to 15% flipped the optimal strategy from "no lockdown" to "permanent full lockdown."

Limitations and Future Work

  • Linear Economic Assumptions: The authors acknowledge that the cost of lockdown is likely non-linear. A 4-week lockdown is often more than four times as damaging as a 1-week lockdown because businesses begin to shutter permanently.
  • Testing Bias: The model assumes an "idealized" testing mechanism and does not account for false positives/negatives or the significant delays in test results that plagued the early pandemic.
  • Spatial Granularity: The current Markovian model treats an entire state as a single unit. Future work aims to integrate the non-Markovian approach to allow for "surgical" lockdowns at the county or city level.

Summary updated

Abstract

We study and predict the evolution of Covid-19 in six US states from the period May 1 through August 31 using a discrete compartment-based model and prescribe active intervention policies, like lockdowns, on the basis of minimizing a loss function, within the broad framework of partially observed Markov decision processes. For each state, Covid-19 data for 40 days (starting from May 1 for two northern states and June 1 for four southern states) are analyzed to estimate the transition probabilities between compartments and other parameters associated with the evolution of the epidemic. These quantities are then used to predict the course of the epidemic in the given state for the next 50 days (test period) under various policy allocations, leading to different values of the loss function over the training horizon. The optimal policy allocation is the one corresponding to the smallest loss. Our analysis shows that none of the six states need lockdowns over the test period, though the no lockdown prescription is to be interpreted with caution: responsible mask use and social distancing of course need to be continued. The caveats involved in modeling epidemic propagation of this sort are discussed at length. A sketch of a non-Markovian formulation of Covid-19 propagation (and more general epidemic propagation) is presented as an attractive avenue for future research in this area.
BibTeX
@misc{eftekhari2020markovian,
  author = {Hamid Eftekhari and Debarghya Mukherjee and Moulinath Banerjee and Ya’acov Ritov},
  title = {{Markovian And Non-Markovian Processes with Active Decision Making Strategies For Addressing The COVID-19 Pandemic}},
  year = {2020},
  eprint = {2008.00375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.00375},
}
Figure