Real Time Forecasting Under Retrospective Data Revisions

This direction is split into other directions: surveillance data backfill. Its last version stays readable and citable here.

Version 1 of 117 papers25 conceptsfrozen 2026-10-04

In the idealized world of machine learning research, "ground truth" is a static target. We download a dataset, split it into training and testing sets, and assume the labels are immutable. However, in high-stakes domains like public health and geopolitical monitoring, the ground truth is a moving target. What we "know" today about how many people had the flu last week will likely change by next month.

This phenomenon, known as backfilling or retrospective revision, creates a fundamental disconnect between how models are trained and how they must perform in the wild.

The Mirage of Finalized Data

Most forecasting models are evaluated on "finalized" datasets—historical records that have been cleaned, audited, and corrected over months or years. As Erkin Ötles and colleagues point out in Mind the Performance Gap: Examining Dataset Shift During Prospective Validation, this creates an "infrastructure shift." Research databases mask the noise and incompleteness of the data available during real-time operations.

When a model is deployed, it doesn't see the finalized version of last week's data; it sees a provisional, often incomplete report. If a model is trained only on clean, finalized data, it learns to rely on a level of precision that simply does not exist during a live crisis. This leads to a significant overestimation of model performance during retrospective testing.

The Surveillance Apparatus as a Variable

To forecast effectively, one must model not just the disease, but the "surveillance apparatus" itself. In What to know before forecasting the flu, Prithwish Chakraborty, Madhav Marathe, and their team argue that understanding how data is collected and revised is often more important than the specific algorithm used.

Public health data is subject to significant reporting lags. An initial report might only capture 60% of actual cases, with the remaining 40% "backfilled" over the following three weeks. A model that treats that initial 60% as the absolute truth will consistently under-predict the severity of an outbreak. Successful teams implement mechanisms to estimate the "final stable value" of a report from its initial, volatile version.

Nowcasting to Bridge the Lag

One way to combat reporting lags is nowcasting: using real-time, non-traditional data signals to estimate the current state of affairs before official reports stabilize.

Mauricio Santillana and Fred Lu have demonstrated this across multiple studies, including Predicting Dengue Incidence Leveraging Internet-Based Data Sources. They found that while historical epidemiological data is a strong predictor when delays are short, internet search trends (like Google Searches) become the primary driver of accuracy when official reports are lagged by a month or more. These "digital proxies" don't suffer from the same administrative backfilling process as clinical reports, providing a more immediate—if noisier—signal of activity.

Operationalizing Instability

The challenge of data instability is not limited to biology. In Real-time 2019 Portuguese Parliament Election Results Dataset, Nuno Moniz highlights how provisional election night figures differ from legally verified results, creating traps for models trained on live streams. Similarly, the DeepCOVID framework, developed by Alexander Rodríguez and others, had to explicitly account for retroactive revisions to remain featured in the CDC’s official ensemble.

The recurring lesson across these papers is that for a model to be useful in real-time, it must be "revision-aware." This involves:

  1. Training on "Raw" History: Evaluating models on the data as it appeared at the time of the forecast, rather than on the finalized version available today.
  2. Uncertainty Quantification: Using methods like the bootstrapping approach in DeepCOVID to communicate the inherent instability of the current signal.
  3. Hybrid Modeling: Combining stable but lagged official data with immediate but noisy proxy data (like mobility or search trends) to balance reliability and timeliness.

By treating the reporting process as part of the system being modeled, researchers can close the gap between laboratory performance and real-world utility.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_real_time_forecasting_under_retrospectiv_38130cda,
  title = {Real Time Forecasting Under Retrospective Data Revisions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-forecasting-under-retrospective-data-revisions/rdr_38130cda20574cd2b73df2ed09a83433},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_real_time_forecasting_under_retrospectiv_38130cda_v1,
  title = {Real Time Forecasting Under Retrospective Data Revisions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-forecasting-under-retrospective-data-revisions/rdr_38130cda20574cd2b73df2ed09a83433/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}