Modeling Data Revisions in Real-Time Epidemic Forecasting

26 concepts15 papers1 proposal

When we look at historical data to evaluate a machine learning model, we are looking at a polished, finalized version of the past. We see exactly how many people were hospitalized with influenza or COVID-19 during any given week three years ago.

But if you were standing in that exact week three years ago, looking at the official dashboard, the number you saw would be vastly different. It would be incomplete, delayed, and subject to weeks of subsequent updates. This phenomenon is known as backfill or retrospective data revision.

In real-time public health forecasting, treating initial data reports as the absolute ground truth is a recipe for failure. To build models that work in the real world, researchers have shifted from treating data as a static time series to modeling the dynamic process of data maturation itself.

The Illusion of Clean Historical Data

Standard machine learning models assume that once a data point is recorded, it is fixed. In epidemiological surveillance, however, the data is a moving target. As Prithwish Chakraborty et al. argue in What to know before forecasting the flu, understanding this "surveillance apparatus"—how data is collected, aggregated, and revised—is often far more important than the specific forecasting algorithm used.

When models are trained and validated on finalized historical datasets, they benefit from hindsight that is unavailable during live deployment. Erkin Ötles et al. formalize this discrepancy as an infrastructure shift in Mind the Performance Gap: Examining Dataset Shift During Prospective Validation. Research databases retroactively store the most accurate, updated values for a given point in time, masking the real-time noise and incompleteness that a model encounters during live inference.

Consequently, models evaluated purely on finalized data suffer a severe performance drop when deployed in real-time operations. This evaluation gap has been documented across various diseases and regions, from state-level influenza nowcasting in the US (Kandula et al.) to city-level dengue tracking in Brazil (Lu et al.).

The Anatomy of a Revision

To understand why data changes, consider how a single health event—like a hospital admission—is recorded.

[Patient Admitted] ➔ [Medical Claim Filed] ➔ [Aggregated by Insurer] ➔ [Reported to Public Health]
       ▲                                                                          │
       └─────────────────── Weeks or Months of Administrative Lag ────────────────┘

A patient enters the hospital. A medical insurance claim is generated, processed, and eventually aggregated. As Roni Rosenfeld notes in Epidemic tracking and forecasting: Lessons learned from a tumultuous year, signals derived from sources like medical claims are prone to pervasive, ongoing revisions over several weeks before reaching a stable state.

We can visualize this data structure not as a simple vector of values over time, but as a two-dimensional matrix:

  • Event Time ($t$): The week the health event actually occurred.
  • Publish Time ($v$): The "vantage date" or the week the data was published.

When a forecaster makes a prediction at week $V$, they only have access to the data slice published up to week $V$. For recent weeks ($t \approx V$), the reported numbers are heavily truncated. For older weeks ($t \ll V$), the numbers have stabilized.

If a model treats the highly incomplete data at the diagonal ($t = V$) as the true state of the epidemic, it will consistently underestimate the current trajectory of a disease outbreak, leading to dangerous delays in public health responses (McGowan et al.; Rodríguez et al.).

How to Forecast the Present

To mitigate this instability, researchers use two primary strategies: bypassing the lag with real-time proxies, or explicitly modeling the revision process.

1. Fusing Auxiliary Real-Time Signals

One approach is to supplement unstable official reports with digital indicators that do not suffer from administrative backfill. In Improved state-level influenza nowcasting in the United States leveraging Internet-based data and network approaches, Fred Lu et al. demonstrate that combining local internet search trends with historical data helps stabilize real-time estimates. When official health reports are delayed or highly volatile, these immediate digital signals act as an anchor, allowing models to "nowcast" the current state of an epidemic before official data matures.

2. Explicitly Learning the Revision Dynamics

Instead of treating backfill as random noise to be avoided, modern frameworks treat it as a systematic, learnable process. Administrative delays are not entirely random; they follow predictable patterns (e.g., weekend backlogs, predictable insurance processing cycles).

In Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future, Harshavardhan Kamarthi et al. introduced a deep learning framework designed to learn these revision patterns. By training on the historical evolution of the data matrix, the framework learns to "correct" preliminary real-time reports, projecting them forward to their final stable values before passing them to a forecasting model.

[Preliminary Data (Unstable)] ➔ [Backfill Model (Learns Revision Patterns)] ➔ [Estimated Stable Value] ➔ [Forecasting Model]

Similarly, Jingjing Tang et al. developed Delphi-RF (Real-time forecasting of data revisions in epidemic surveillance streams), which uses nonparametric quantile regression to efficiently predict the distribution of future data updates. This allows public health officials to see not just a single noisy data point, but a probabilistic distribution of what that data point will eventually become once all claims are settled.

Why Revision Modeling Works

Explicitly modeling data revisions works because it aligns the training environment with operational reality. Platforms like the COVIDcast API (Reinhart et al.) make this possible by meticulously tracking and preserving every historical revision of public health indicators.

When a model is trained on these "vantage date" snapshots rather than finalized data, it learns to recognize the systematic under-reporting characteristic of recent weeks. By transforming the problem from "predicting the future based on perfect data" to "predicting the future based on predictably imperfect data," these frameworks close the performance gap, providing reliable situational awareness when it matters most.

Key Foundational Work

What to know before forecasting the flu (2018) — contributed 3 concept elements to this direction.
Subregional Nowcasts of Seasonal Influenza Using Search Trends (2017) — contributed 1 concept elements to this direction.
Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_surveillance_data_backfill_6393fc5f,
  title = {surveillance data backfill},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surveillance-data-backfill/rdr_6393fc5f9d2b4c01af01fc9515ea04a5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_surveillance_data_backfill_6393fc5f_v1,
  title = {surveillance data backfill},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surveillance-data-backfill/rdr_6393fc5f9d2b4c01af01fc9515ea04a5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake