How Much of Search-Based Disease Forecasting Survives an Honest Backtest?

Search-based disease surveillance tries to predict how many people are sick right now, or will be next week, from what the public types into search engines. People search for "sputum" or "runny nose" days before they see a doctor, and official case counts arrive weeks after that. The evidence for the idea comes almost entirely from retrospective backtests: researchers download several years of search volume and case counts, select the queries that track cases best, and show that a forecaster fed only those queries beats one fed past case counts. This note argues that these backtests carry several forms of hidden future information. Four are specific to search data: choosing keywords using the whole evaluation period, rescaling search volume against a peak that lies in the future, scoring against case counts that were revised long after the fact, and generating keyword lists with a language model trained on text from the evaluation period. A fifth is general: the forecasting model may have been pretrained on the public evaluation series. The proposal measures how much of the published advantage of search signals each of these is responsible for. It then replaces the retrospective backtest with a prospective, delay-aware protocol that makes such leakage impossible by construction.

The second part is deliberately low-tech. When the target is a future epidemic week, the "test set" does not exist yet, so nothing needs to be encrypted. Forecasters only need to publish a cryptographic commitment to their forecast before the truth is generated. A commitment is a hash that fixes the forecast without revealing it, and it is opened later. Encryption still matters, but in a narrower place: it protects the archived "as-of" snapshots that make honest retrospective work possible, so that those snapshots do not themselves become pretraining data.

Core Claim

Our claim is that published search-only forecasting gains are an upper bound inflated by look-ahead, and that the size of the inflation can be measured and broken into parts. We take the Korean rhinovirus study, Can Symptom Search Trends Forecast Respiratory Virus Patient Counts? A Rhinovirus Case Study in Korean Syndromic Surveillance, as the anchor. It shows a DLinear model on Naver "sputum" searches reaching an error (root mean squared error, RMSE) of 82.3 against 95.7 for a model on patient history alone. Its keywords were filtered by cross-correlation over the full 270-week window. We expect part of that gap to disappear when keywords are chosen only from past data and search volume is normalized as of each forecast date.

Two other leaks pull in opposite directions. Reporting delay, the weeks before official counts become visible, hurts the patient-history baseline and so should widen the gap in favor of search. The industrial-delays study shows that ignoring delay can misstate performance by up to 39.86% Matthews Correlation Coefficient (MCC) in financial pipelines. Because these effects cancel partly, the net honest advantage of search is a genuinely open empirical question, not a foregone debunking. That two-sided uncertainty is what makes the study worth doing.

Research Program

The first component is a leakage decomposition. We rebuild the forecasting pipeline so that each source of future information can be switched on or off independently.

Keyword selection is the first switch. It is redone inside a rolling window: at forecast date $T$, candidate queries are ranked using correlations computed only on weeks before $T$ minus the reporting delay.

Normalization is the second. Relative search volume (RSV) is scaled against the maximum in whatever window is downloaded. A single 2019–2024 download therefore rescales 2020 values using peaks from later years. We rebuild point-in-time series from expanding-window downloads, or failing that by rescaling each prefix of the series against its own past maximum.

The lexicon generator is the third. The Korean study used GPT-4o to expand clinical terms into 3,336 colloquial queries. A model trained on 2020–2024 web text may already "know" which symptoms trended during COVID-era respiratory seasons. We compare that lexicon with one built from a fixed pre-2019 medical synonym source, and with one produced by an older open model whose training cutoff precedes the window.

Delay and revision form the fourth switch. Ground truth is shifted so that at time $T$ only counts through $T-\Delta$ are visible, with $\Delta$ set to the sentinel reporting lag. Where archived vintages exist, as for CDC influenza-like illness (ILI) reports, scoring uses the count that was actually published at the decision time. Where they do not exist, revision is simulated by adding noise to the most recent weeks, which tests sensitivity rather than claiming realism.

The fifth switch addresses pretrained time-series foundation models such as TimesFM, Moirai and Chronos. Their training corpora may include public surveillance series. We test this by comparing each model's zero-shot error on the real series against its error on matched surrogates. The surrogates are real series time-shifted by a year, and synthetic SIRS trajectories (a standard model of epidemic spread in which people move between susceptible, infected, recovered and back to susceptible) fitted to match the real series' seasonality. A real-minus-surrogate gap that is larger for foundation models than for models trained from scratch is evidence of memorization. Following Test of Time, which showed that surface perturbation can erase the post-cutoff accuracy drop while memorized knowledge persists, we treat a missing gap as weak evidence of cleanliness and a present gap as strong evidence of contamination.

The second component is a prospective protocol run for at least one respiratory season on the Korean and U.S. ILI targets. Each week, participants publish a hash of their point and quantile forecasts for horizons beyond the current reporting frontier. The hashes are opened only after the official count for that week is released. Scoring is delay-aware in the sense of the industrial-delays study: it uses only data that was visible at commitment time.

The weekly as-of snapshots are released as encrypted archives with a public password and a no-derivatives license, as advocated by Stop Uploading Test Data in Plain Text. These snapshots are the search-volume downloads and the case counts as published that week. This adds friction against crawlers while keeping the snapshots usable for later point-in-time research. That paper's audit, which found about 4.2 unofficial evaluations for every official leaderboard submission, is our motivation for not posting snapshots in plain text. We accept its implication that friction delays leakage rather than preventing it, which is exactly why the live evaluation rests on commitments rather than secrecy.

Baselines

The naive reference points are seasonal persistence, which forecasts the same week from last year, and last-observed-value persistence under delay, which forecasts the most recent visible count. The Impermanent live benchmark found SeasonalNaive competitive on MASE (mean absolute scaled error), so any search model that cannot beat these is not useful in deployment.

The patient-history baselines are the same backbones as in the Korean study, DLinear and GRU, trained on case history alone. Their reported RMSEs are 95.7 for DLinear and 64.9 for GRU (Korean study). We rerun them with identical splits, an identical hyperparameter search budget, and the same delay $\Delta$. They are the strongest fair competitor because they represent what a health agency can already do without search data. We also run AutoARIMA and AutoETS, which Impermanent found better calibrated than several deep models.

The strongest existing search method is the published leaky pipeline itself: Naver "sputum" with DLinear at RMSE 82.3, and Google "rhinitis" with GRU at RMSE 62.9. We first reproduce these numbers within tolerance before applying any ablation. For U.S. ILI, the strongest hybrid forecaster in the file is EpiNode (How (Not) to Hybridize Neural and Mechanistic Models), which reports "up to a 57% reduction" in RMSE over its baselines. No absolute RMSE for it is reported on Lacuna, so we rerun it ourselves with and without search inputs.

The ablation that isolates our key ingredient is the leak-free pipeline: identical model and data, with each leak switched off one at a time and then all together. The proposal "wins" in two ways. First, if the leak-free search model still beats the delay-matched patient-history baseline on RMSE and on turning-point lead time, with a block-bootstrap confidence interval that excludes zero, the search hypothesis survives an honest test. Second, if it does not, the decomposition tells the field which leak manufactured the published gain. Either outcome is informative. The study fails only if we cannot reproduce the published numbers in the first place.

Why not just…?

Why not just use a single chronological train/test split? A global cutoff stops training on future weeks, but it does nothing about keywords selected over the full window, RSV rescaled against future peaks, or an LLM lexicon written with knowledge of the test years. The Korean study's leakage enters before the split is ever made. A single frozen test period also hides how performance shifts across seasons, the effect LiveHouse-TS documents as ranking inversion between static and live evaluation. We include a global-split arm so that readers can see exactly how much it misses.

Why not just build a zero-knowledge benchmark like P⁴Bench? P⁴Bench proves that a model knows an answer satisfying an exact relation, such as a SAT assignment, without revealing it. Forecast error on a real-valued count has no such relation to prove. More fundamentally, ZK protection is unnecessary when the target lies in the future, because a hash commitment already guarantees the forecast was made blind. P⁴Bench also concedes that a proof cannot show which system produced the answer, and commitments share that limitation. We acknowledge this openly; it limits claims about which model made a forecast, not about whether the forecast saw the truth.

Why not just detect contamination by checking for accuracy decay after the training cutoff? Test of Time shows that this signal is fragile: rephrasing turned a -3.83-point drop into a +2.19-point gain, and influence tracing fell from 77.5% to 17.5%. Its central lesson is that decay is phrasing-dependent and unreliable as a contamination signal in either direction. In our setting, the leaks in keyword selection and normalization are not model memorization at all, so no decay test would ever see them.

Why not just fix brittleness with a regularized multi-keyword ensemble? This is a real alternative. A ridge or sparse model over the whole lexicon removes the hand-picked single query and could recover much of the honest gain. We therefore include it as an arm, fitted strictly on past data. If it matches the leaky single-keyword result, the practical recommendation becomes "use regularized ensembles with as-of normalization." That would be a useful positive result and not a threat to the study.

Proposed Experiments

The primary experiment uses the KDCA rhinovirus, parainfluenza and respiratory syncytial virus series with Naver DataLab and Google Trends inputs, October 2019 to December 2024. It compares the published leaky pipeline, each single-leak ablation, and the fully leak-free pipeline across the 20 backbones, reporting RMSE, MASE, and weeks of lead at epidemic turning points. Probabilistic models are additionally scored with CRPS (continuous ranked probability score, a calibration-aware error for forecast distributions).

The replication experiment repeats the analysis on CDC ILI across the ten HHS regions for 2022–2025, using Google Trends inputs and archived report vintages. The contamination experiment runs the real-versus-surrogate test for TimesFM, Moirai and Chronos against from-scratch DLinear on both datasets. The prospective experiment runs the commitment protocol for one season and checks whether retrospective leak-free rankings predict live rankings better than leaky ones do. This is measured by rank correlation between each retrospective ranking and the live ranking.

The decisive check has two parts. The first is whether the fully leak-free, delay-aware search model still beats the delay-matched patient-history baseline. The second is whether the leak-free retrospective ranking agrees with the live ranking more closely than the published-style ranking does. If both hold, honest backtests can be trusted as a proxy for deployment, and search signals earn their place.