Bayesian imputation of COVID-19 positive test counts for nowcasting under reporting lag

Radka Jersakova, James W. Lomax, James Hetherington, Brieuc Lehmann, George Nicholson, Mark Briers, Chris Holmes

TL;DR

This paper introduces a Bayesian hierarchical model to "nowcast" COVID-19 case counts in the UK, correcting for the inherent 4-5 day reporting lag in test results. By combining binomial thinning with a latent intensity process and Sequential Monte Carlo inference, the authors provide real-time estimates of infection trends with quantified uncertainty, allowing for faster and more reliable public health decision-making.

What Problem Does This Paper Address?

In the heat of an epidemic, timing is everything. However, official "Pillar 2" (symptomatic) COVID-19 test counts in the UK suffer from a significant reporting lag. It often takes up to five days for a positive test result to be processed, centralized, and reported.

This creates a "fog of war" for policymakers: the data for the most recent few days always looks like cases are dropping, simply because the results haven't arrived yet. Standard moving averages often ignore these incomplete days, leaving a multi-day gap in situational awareness. The authors aim to fill this gap by nowcasting—statistically imputing the final counts based on the partial information available today.

Methods

The authors propose a Bayesian hierarchical model that treats the "true" unobserved count as a latent variable. The model consists of three main components:

  1. Binomial Thinning for Reporting Lag: They model the reported count as a binomial draw from the true count. The "success probability" (reporting rate) is estimated using empirical priors. They found that reporting behavior is often spatially stable (neighbors report similarly) or temporally stable (an area reports similarly to its own recent past).
  2. Latent Intensity Process: To ensure the "true" counts make sense over time, they model an underlying infection intensity ($\lambda$) using a Poisson process. They impose a random walk prior with drift to enforce local temporal smoothness, which is more flexible than the cubic splines used in similar works.
  3. Weekend Effects: Real-world data shows predictable dips on Saturdays and Sundays due to reduced testing capacity. The model incorporates a latent variable ($z_t$) to "soak up" these periodic drops, preventing the model from mistakenly thinking the infection rate itself has decreased.

Inference: Because the model is non-conjugate and non-linear, they use Sequential Monte Carlo (SMC)—specifically a forward-backward particle filtering and smoothing algorithm—to generate the posterior distribution of the true counts.

Datasets

The study utilizes publicly available UK COVID-19 data from the official government dashboard (coronavirus.data.gov.uk).

  • Granularity: Data is analyzed at the Lower-Tier Local Authority (LTLA) level.
  • Scope: There are 316 LTLAs in England.
  • Timeframe: The paper specifically highlights examples from late 2020 (October–December), a period of high volatility as the UK transitioned between lockdowns.

Key Insights and Ideas

  • The "Sufficient Statistic" Shortcut: The authors demonstrate that the most recent available report for a specific date is a "sufficient statistic" for the true count. This simplifies the math significantly, allowing them to ignore the intermediate sequence of updates and focus only on the latest "under-report."
  • Predictable Lag: One of the most important findings is that reporting lag isn't random noise; it’s highly predictable. By "learning" the thinning rates from the previous two weeks of converged data, the model can accurately guess how much of the "iceberg" is still underwater.
  • Stability Modes: The paper identifies that when an area's reporting becomes erratic (temporally unstable), you can often "borrow" reporting rate estimates from geographical neighbors (spatial stability) to maintain accuracy.

Key Results

  • Reduced Error: The model significantly outperforms the baseline strategy of simply ignoring the most recent (incomplete) days. As shown in their evaluation (Figure 15), the Mean Absolute Error (MAE) and the variance of the estimates are much lower than the baseline across different lag periods.
  • Probabilistic Alerting: Unlike a simple moving average, this model provides a full posterior distribution. This allows for an "alerting system" where authorities can calculate the exact probability that the infection rate has crossed a specific threshold (e.g., "There is a 90% probability that cases in Leeds are now above 100 per day").
  • Smoothing without Lag: By accounting for weekend effects, the latent intensity ($\lambda$) remains smooth even when the raw reported data zig-zags, providing a clearer picture of the epidemic's true trajectory.

Limitations and Future Work

  • Prior Stability: The model relies on the reporting process remaining stable. If a laboratory has a massive technical failure (as happened in late September 2020), the model’s priors will be incorrect until it "re-learns" the new (broken) behavior.
  • Lab-Level Modeling: Currently, the model operates at the LTLA level. However, LTLAs are served by different labs. The authors suggest that modeling the lag at the testing laboratory level (using available capacity data) could provide even tighter priors.
  • Spatial Coupling: While they use spatial info for priors, the core latent intensity model doesn't currently "link" neighboring LTLAs. Incorporating a Gaussian Markov Random Field (GMRF) could improve results by accounting for the fact that viruses don't respect administrative boundaries.

Summary updated

Abstract

Obtaining up to date information on the number of UK COVID-19 regional infections is hampered by the reporting lag in positive test results for people with COVID-19 symptoms. In the UK, for "Pillar 2" swab tests for those showing symptoms, it can take up to five days for results to be collated. We make use of the stability of the under reporting process over time to motivate a statistical temporal model that infers the final total count given the partial count information as it arrives. We adopt a Bayesian approach that provides for subjective priors on parameters and a hierarchical structure for an underlying latent intensity process for the infection counts. This results in a smoothed time-series representation now-casting the expected number of daily counts of positive tests with uncertainty bands that can be used to aid decision making. Inference is performed using sequential Monte Carlo.
BibTeX
@article{jersakova2022bayesian,
  author = {Radka Jersakova and James W. Lomax and James Hetherington and Brieuc Lehmann and George Nicholson and Mark Briers and Chris Holmes},
  title = {{Bayesian imputation of COVID-19 positive test counts for nowcasting under reporting lag}},
  journal = {Journal of the Royal Statistical Society Series C (Applied Statistics)},
  year = {2022},
  doi = {10.1111/rssc.12557},
}
Figure