Nonsmooth Convex Optimization to Estimate the Covid-19 Reproduction Number Space-Time Evolution With Robustness Against Low Quality Data

Barbara Pascal, Patrice Abry, Nelly Pustelnik, Stéphane G. Roux, Rémi Gribonval, Patrick FlandrinIEEE TSP 2022 · OpenReview

TL;DR

This paper introduces a novel optimization framework to estimate the COVID-19 reproduction number (R) over space and time, specifically designed to be robust against the notoriously low-quality daily infection data. By jointly estimating R and "outliers" that model data imperfections, and incorporating epidemiological constraints, their method provides more reliable and smoother estimates crucial for effective pandemic monitoring.

What Problem Does This Paper Address?

Accurately tracking the spread of infectious diseases like COVID-19 hinges on estimating the reproduction number (R), which indicates how many new infections an infected person causes. This is vital for public health authorities to design and implement effective counter-measures. However, the real-world data used for this estimation – daily new infection counts – is often plagued by issues like missing data, incorrect counts, and artificial dips due to reporting schedules (e.g., weekends). These data quality problems severely impair the accuracy of traditional R estimation methods, making timely and reliable pandemic assessment a significant challenge. The authors aim to overcome this by developing a method that can handle these "noisy" data points gracefully.

Methods

The core of the paper is a new optimization problem formulation. They extend their previous work by:

  • Jointly Estimating R and Outliers: Instead of trying to clean the data beforehand, they model "outliers" directly within the optimization. These outliers capture the deviations from expected behavior caused by data quality issues.
  • Crafting a Robust Functional: They define a functional to minimize that combines:
  • Data-Model Fidelity: A likelihood term based on a Poisson model for daily infections, but extended to account for the estimated outliers.
  • Epidemiological Regularity: Constraints that enforce R to be smooth over time (piecewise linear) and spatially correlated across connected regions. This reflects the biological reality of disease spread.
  • Sparsity for Outliers: A constraint that assumes outliers are infrequent, meaning only a few data points are truly problematic.
  • Convex Optimization: Crucially, they ensure the entire functional remains convex. This guarantees that a unique minimum exists and can be found efficiently.
  • Proximal Algorithms: They develop and theoretically analyze a primal-dual proximal algorithm to solve this non-smooth convex optimization problem. This algorithm is designed for efficient, iterative updates.

Datasets

The authors evaluate their method using two real-world COVID-19 datasets:

  • Dataset 1 (Country-level): Daily new infection counts for over 200 countries, sourced from Johns Hopkins University. This dataset covers the period from February 15th, 2020, to August 31st, 2021.
  • Dataset 2 (French Counties): Daily hospital-recorded new infection counts for the 96 départements (counties) of metropolitan France, provided by Santé-Publique-France. This dataset covers March 19th, 2020, to August 31st, 2021.

These datasets are publicly available and represent real-world challenges in pandemic data collection.

Key Insights and Ideas

  • Outliers as a Feature, Not a Bug: The most significant insight is treating data quality issues (outliers) as an integral part of the estimation problem, rather than something to be removed beforehand. This allows the model to adapt to imperfect data.
  • Joint Estimation is Key: Estimating R and outliers simultaneously, within a single optimization framework, leads to more accurate and smoother R estimates compared to two-step approaches.
  • Epidemiological Constraints Matter: Incorporating prior knowledge about how diseases spread (temporal smoothness, spatial correlation) significantly improves the reliability and interpretability of R estimates, especially with noisy data.
  • Convexity Enables Efficiency: By carefully constructing the objective function to be convex, the authors pave the way for provably convergent and efficient algorithms, making daily updates feasible even for large datasets.

Key Results

  • Robustness Demonstrated: The proposed method (denoted as $R_o$) consistently produces smoother and more reliable estimates of R compared to methods that either ignore outliers or use a two-step cleaning process. This is visually demonstrated in figures comparing their estimates against others, showing significantly less spurious variability.
  • Accurate Outlier Detection: The method effectively identifies and accounts for data anomalies, such as pseudo-seasonal reporting effects (e.g., lower counts on weekends) and other count misreports.
  • Spatial Correlation Benefits: For regional data (French départements), incorporating spatial regularization ($R_{o,s}$) further enhances the smoothness and realism of R estimates, reflecting the interconnectedness of neighboring areas.
  • Efficient Updates: The developed primal-dual algorithm is efficient enough to allow for automated daily updates of R estimates for hundreds of countries and regions, with results made available via interactive maps.

Limitations and Future Work

  • Hyperparameter Tuning: While the authors propose a data-driven approach for hyperparameter selection, it's noted that this can still be time-consuming, especially when applied to numerous countries with varying characteristics.
  • Uniqueness of Minimizer: The paper acknowledges that while a minimizer is guaranteed to exist, the strict convexity of the overall functional is not proven, meaning the uniqueness of the R estimate itself (though the Poisson parameter p is shown to be unique) is not definitively established.
  • Future Work: The authors suggest future research directions including:
  • Developing more automated, adaptive hyperparameter selection strategies.
  • Constructing confidence intervals for the R estimates.
  • Extending the method for forecasting.

Summary updated

Abstract

Daily pandemic surveillance, often achieved through the estimation of the reproduction number, constitutes a critical challenge for national health authorities to design counter-measures. In an earlier work, we proposed to formulate the estimation of the reproduction number as an optimization problem, combining data-model fidelity and space-time regularity constraints, solved by nonsmooth convex proximal minimizations. Though promising, that first formulation significantly lacks robustness against the Covid-19 data low quality (irrelevant or missing counts, pseudo-seasonalities,...) stemming from the emergency and crisis context, which significantly impairs accurate pandemic evolution assessments. The present work aims to overcome these limitations by carefully crafting a functional permitting to estimate jointly, in a single step, the reproduction number and outliers defined to model low quality data. This functional also enforces epidemiology-driven regularity properties for the reproduction number estimates, while preserving convexity, thus permitting the design of efficient minimization algorithms, based on proximity operators that are derived analytically. The explicit convergence of the proposed algorithm is proven theoretically. Its relevance is quantified on real Covid-19 data, consisting of daily new infection counts for 200+ countries and for the 96 metropolitan France counties, publicly available at Johns Hopkins University and Santé-Publique-France. The procedure permits automated daily updates of these estimates, reported via animated and interactive maps. Open-source estimation procedures will be made publicly available.
BibTeX
@article{pascal2022nonsmooth,
  author = {Barbara Pascal and Patrice Abry and Nelly Pustelnik and Stéphane G. Roux and Rémi Gribonval and Patrick Flandrin},
  title = {{Nonsmooth Convex Optimization to Estimate the Covid-19 Reproduction Number Space-Time Evolution With Robustness Against Low Quality Data}},
  journal = {IEEE Trans. Signal Process.},
  year = {2022},
  url = {https://openreview.net/forum?id=SVh9js6vbZ},
}
Figure