Accounting for incomplete testing in the estimation of epidemic parameters

Rebecca A. Betensky, Yang Feng

This paper, published in the early months of the COVID-19 pandemic, addresses a fundamental flaw in how we track outbreaks: our reliance on "crude" case counts that are inextricably linked to how much testing we are actually doing.

TL;DR

Researchers from NYU demonstrate that standard epidemic metrics, like doubling time, are heavily biased by fluctuating testing strategies. By applying Bayes' theorem and using death counts as a more stable proxy for true infections, they provide a framework to adjust these estimates and perform sensitivity analyses to uncover the "true" state of an epidemic.

What Problem Does This Paper Address?

In the early stages of an epidemic, public health officials rely on the "epidemic curve"—a plot of confirmed cases over time—to decide if interventions like social distancing are working. However, this curve is a "foggy lens." If the number of tests conducted drops or fails to keep pace with the virus, the curve might flatten even if the actual infection rate is skyrocketing.

The authors focus on doubling time ($T_d$), the time it takes for the total number of cases to double. While $T_d$ is a more intuitive measure of growth than raw counts, it is still "intertwined" with testing. If you test less today than you did last week, your estimated doubling time will look longer (more optimistic) than it actually is. The paper seeks to quantify this relationship and provide a way to correct the bias.

Methods

The authors use a non-parametric approach rooted in Bayes' Theorem to relate the observed number of cases ($O(t)$) to the true, unknown number of infections ($C(t)$).

  1. The Testing Ratio ($R$): They define a ratio, $R(t, d)$, which represents the change in the probability of an infected person being tested between two time points. If $R=1$, testing is stable. If $R > 1$, testing has decreased, leading to an overestimation of doubling time (false optimism).
  2. Deaths as a Proxy: Since the "lethality" (Infection Fatality Rate) of a virus is generally constant over short periods, the authors suggest that the ratio of deaths to true cases should be stable. By comparing the ratio of deaths to observed cases at different times, they can estimate the testing ratio $R$.
  3. The Lag Factor: They incorporate a lag time ($l$)—specifically 18 days for COVID-19—to account for the duration between a positive test (or symptom onset) and death.
  4. Sensitivity Analysis: For regions where death data is unreliable, they propose a sensitivity analysis where researchers plug in hypothetical decreases in testing (e.g., 10%, 20%, 25%) to see how much the doubling time estimate would shift.

Datasets

The study utilized two primary data sources from the early 2020 period:

  • The COVID Tracking Project (covidtracking.com): Used for positive test counts across the 12 US states with the highest caseloads as of April 7, 2020 (including NY, NJ, CA, and WA).
  • NYC Health Department (GitHub): Used for "probable and confirmed" death data in New York City to estimate the testing ratio $R$ specifically for that region.

Key Insights and Ideas

  • The "Testing Trap": The paper mathematically proves that if testing probability decreases, the crude doubling time is an overestimate. This means the epidemic is growing faster than it appears on the news.
  • NYC was a Counter-Example: Interestingly, in New York City during late March and early April 2020, the data suggested that testing of infected individuals actually increased slightly. Consequently, crude estimates for NYC were actually too pessimistic; the epidemic was slowing down slightly faster than the raw case numbers suggested.
  • Stability is Key: The authors argue that doubling time should be the primary measure for public health, but only when adjusted for testing dynamics.

Key Results

  • Bias Magnitude: The authors found that failing to account for a 25% drop in testing probability could lead to an overestimation of doubling time by as much as three days. In a fast-moving pandemic, a three-day error in growth rate is massive for policy planning.
  • Visualizing Sensitivity: The paper provides a series of "spaghetti plots" (Figures 1-3) for 12 states, showing how doubling time estimates diverge wildly depending on whether you assume testing is stable or declining.
  • NYC Adjustment: For New York City, the calculated $R(t, d)$ was often less than one, indicating that the "true" doubling time was actually longer (better) than the "observed" one.

Limitations and Future Work

  • The Retrospective Lag: The biggest limitation is the 18-day lag required for death data. This means the "death proxy" method can tell you what the testing situation was three weeks ago, but it cannot provide a real-time correction for today's numbers.
  • Data Quality: The method relies on the accurate reporting of "probable" deaths. In many jurisdictions, only deaths with a confirmed lab test were recorded, which re-introduces the very testing bias the authors are trying to avoid.
  • Resource Constraints: The assumption that lethality is constant might break down if a healthcare system is overwhelmed (e.g., lack of ventilators), which would cause the death rate to spike independently of the infection rate.

Summary updated

Abstract

0.2 0.4 0.6 0.8 1.0 u: days prior to time t R(t,u): ratio of probability of testing infected
BibTeX
@article{betensky2021accounting,
  author = {Rebecca A. Betensky and Yang Feng},
  title = {{Accounting for incomplete testing in the estimation of epidemic parameters}},
  journal = {International Journal of Epidemiology},
  year = {2021},
  url = {https://openreview.net/forum?id=uf2R1WpInt},
}

Loading related content...

Figure