Backfill Revisions as a Memorization Test for Time-Series Foundation Models

Many public time series are revised after they are first published. Case counts get backfilled, and economic indicators get corrected. So the same stretch of history exists in two versions: the numbers as first released, and the numbers as finally settled. We propose to use this ordinary fact of data engineering as a built-in experiment for detecting memorization in forecasting models. Give a model the context exactly as it was first released. A model that learned general dynamics should respond to the numbers it is shown. A model that memorized the finalized version from its training corpus will be pulled toward values it could not legitimately have known. Because both versions are timestamped by the data publisher, we know which version was public when, without having to guess at training corpora.

The second idea concerns where to look. Memorization in a forecaster is unlikely to be spread evenly across a series. It should concentrate at distinctive local events such as a peak, a turning point, or a sharp drop. These are exactly the events that shapelets capture. A shapelet is a short characteristic subsequence, matched anywhere in a longer series by its minimum sliding-window distance. Global error metrics average such events away over long, uneventful stretches. We therefore measure the pull toward the finalized version separately at shapelet-matched windows, and report which shape primitives a model has memorized. We treat this localization as a hypothesis to be tested against a plain global version of the same test, not as an assumption.

Core Claim

The core claim has two parts. First, the gap between first-release and finalized data provides a calibrated, attributable contamination test for time-series models. It does not depend on training cutoff dates, prompt formats, or access to training data. Second, measuring that test at shapelet-matched local windows detects memorization with fewer probe series than the global version, because memorized content concentrates at distinctive motifs.

The first part rests on the versioning literature. The second is the proposal's riskiest ingredient, and the experiments are designed so that it can fail cleanly.

Where This Sits

The versioning side supplies the instrument. Jensen's account of bitemporal modeling, in Letter from the Impact Award Winner, separates two timestamps. One is valid time, when an event happened. The other is transaction time, when the database recorded or changed the record. The COVIDcast repository (An open repository of real-time COVID-19 indicators) implements both timestamps for more than 170 signals, as time_value and issue_date. It reports that finalized values can differ from initial reports by over 10%. It also shows that evaluating on finalized data inflates apparent skill.

COVIDcast treats this inflation as a hindsight problem for forecasters who train on finalized data. We turn it around. If a model's pretraining corpus contained finalized data, the gap between versions becomes a signature of what the model absorbed.

The contamination side motivates the work and also constrains it. Recall Residualisation prompts language models to recall public series such as FRED UNRATE and Fama–French Mkt-RF from their dates. It then regresses that recall out of the forecasts with ordinary least squares. The authors report that apparent correlations as high as 0.93 fall to 0.17 or lower, and that LeakShare (the fraction of skill due to recall) is at least 0.97 in 19 of 20 settings. The same paper finds zero parametric recall for numeric time-series foundation models such as Chronos and TimesFM, which perform on par with an AR(1) baseline.

That negative result is the one our proposal must confront directly. Their probe asks a model to recall a series from its date. A numeric model receives no dates, so this probe cannot fire for it by construction. Our probe asks a different question: given the context window, does the model complete it the way the finalized version did? That is conditional memorization. Their test is not designed to see it, and our test can. If our calibrated test also finds nothing in these models, that is a stronger confirmation of their conclusion. If it finds something, it qualifies their conclusion.

Test of Time constrains the design in another way. It shows that drops in performance after a training cutoff disappear when content is rephrased. Influence tracing also falls from a 77.5% hit rate to 17.5% under semantic transformation. We take two lessons from this. Cutoff-based signals are unreliable, so we do not use them. And any probe must survive the transformations that models themselves apply to inputs. For numeric forecasters those transformations are mainly rescaling and offset normalization.

This is where shapelets become more than decoration. VQShape (Abstracted Shapes as Tokens) decomposes subsequences into scale- and offset-invariant shape tokens drawn from a codebook of 64 codes. Those codes cluster into roughly 60 groups shared across domains. It reports a mean accuracy of 0.723 on 29 UEA datasets. Its invariances match what a normalizing forecaster could plausibly memorize: shape, not level. We reuse its codebook, and a ShapeNet-style shapelet extractor (ShapeNet), as the vocabulary for localizing the probe.

Two live or partitioned approaches are relevant as neighbors rather than as the method. Impermanent evaluates forecasters prequentially on live GitHub activity, meaning each forecast is made before the data arrives. Time Machine GPT pretrains a separate model on each yearly text slice. Both avoid contamination rather than measure it. Neither can audit an already-released model on historical data.

The Test

Fix a forecast origin date. Let $x^{I}$ be the context window as it was issued on that date, and let $y^{I}$ and $y^{F}$ be the first-release and finalized values of the target horizon. Define the revision as $\delta = y^{F} - y^{I}$.

Some revision is legitimately predictable, because backfill follows regular reporting patterns. We therefore fit a revision nowcaster $\hat{r}$ using only revision histories issued before the origin date. We then regress the model's excess movement on the unpredictable part of the revision:

$$f(x^{I}) - y^{I} - \hat{r} = \beta\,(\delta - \hat{r}) + \varepsilon,$$

Here $f$ is the forecaster and $\beta$ is the leak coefficient. A value of $\beta \approx 0$ means the model tracks only what was knowable. A value of $\beta \approx 1$ means it reproduces the finalized version.

For probabilistic forecasters we add a companion statistic. This is the log-likelihood ratio of the finalized versus first-release context, compared against series from the same source whose finalized versions were published after the model's release.

The localized variant estimates $\beta$ separately for horizons whose finalized trajectory matches a shapelet or shape token within a distance threshold, and for all remaining horizons. It produces a per-token leak profile that answers which kinds of local events were memorized.

Research Program

The program has three stages.

The first stage builds the data layer. It is a bitemporal store of COVIDcast signals, organized as a base snapshot plus ordered, issue-dated deltas, so that any as-of state can be rebuilt deterministically. This store is the reusable artifact.

The second stage is a controlled pretraining study, which is the only place where memorization ground truth is known. We pretrain small decoder-only forecasters, at roughly the 117M-parameter scale that Time Machine GPT used for text. The corpus combines public time-series data with COVIDcast series, randomly assigned by region to three conditions: finalized version included, first-release version included, or excluded. Each included series is duplicated at a controlled count. This gives exact labels for which region-signal pairs were memorized, and in which version. It lets us measure the detection power of $\beta$ against duplication count, both globally and in the localized variant.

The third stage applies the calibrated test to released models. These are Chronos, TimesFM, and MOMENT as numeric models, plus a text-serialized language-model forecaster in the style of Recall Residualisation. The language-model forecaster serves as a positive control, where memorization is already expected.

Baselines

The naive reference point is standard evaluation. It computes forecast error against finalized data and reads that error as skill. It is what most benchmarks do, and it defines the inflation we are trying to explain.

The first real baseline is as-of evaluation in the COVIDcast manner: score forecasts against the data available at the origin date. It exposes a gap but cannot attribute that gap to memorization, since legitimate learning of backfill patterns produces the same gap. Our nowcaster-controlled $\beta$ is meant to separate the two.

The strongest existing contamination method is recall residualisation from Recall Residualisation. Its reported results are that apparent correlations as high as 0.93 fall to 0.17 or lower, and that LeakShare is at least 0.97 in 19 of 20 language-model settings. We will run it on the same series, forecast origins, and models, with its direct-recall and elicitation prompts applied to the language-model forecaster. For numeric models we will implement its closest analogue, since those models cannot be prompted by date.

Beating it means one of two things. The first is detecting contamination in controlled pretrained numeric models that residualisation scores as clean. The second is matching its verdicts on the language-model positive control while additionally reporting where the leakage sits.

Two ablations isolate the key ingredient. The global-$\beta$ ablation drops all shapelet localization. A random-window ablation localizes to windows of the same lengths and counts, but not matched to shapelets. The localized test counts as beating both only if it raises detection AUC at a fixed 5% false-positive rate in the controlled study, at matched numbers of probe series. The comparison uses the same models, the same forecast origins, and the same nowcaster for all variants. The shapelet threshold is tuned only on a held-out set of controlled regions.

Why not just…?

Why not just evaluate on live data, as Impermanent does? Live prequential evaluation (forecasting each point before it is observed) prevents future contamination. It says nothing about models already trained on history. It also cannot audit past regimes such as the pandemic, which is when revisions were largest. We will use Impermanent's GitHub streams as a negative control: data published after a model's release cannot have been memorized, so $\beta$ there should be near zero. If it is not, our test is biased.

Why not just use recall residualisation or training-cutoff decay? Cutoff decay is fragile under transformation, as Test of Time shows. Date-prompted recall has no handle on numeric models that never see dates. Residualisation is also linear in the recall signal. We will nonetheless report it side by side, and the controlled study will show directly whether it misses memorization that we planted.

Why not just run the global version test without shapelets? This is the real threat. The revision itself already marks where the versions differ. So a global $\beta$, or a plain likelihood-ratio membership test, may be just as powerful. The decisive experiment is the head-to-head detection-power curve in the controlled study.

If the localized variant does not improve AUC at matched probe budgets, we will say so. The contribution then reduces to the version-gap test and the per-token leak profile. The profile would remain useful for explanation but would not be justified by detection power. We expect localization to help mainly when the input normalization used by Chronos-style models erases differences in level. The global test should weaken most in exactly that condition, so we will run the comparison with inputs both normalized and unnormalized.

Why not just fine-tune released models on as-of data to remove the leak? Time Machine GPT reports that temporally adapted text models retain knowledge of the future after fine-tuning on past slices. We will test whether the same holds numerically by fine-tuning a controlled model on first-release data and re-measuring $\beta$. Persistence of the leak after fine-tuning would support auditing over patching.

Decisive Checks

Four results would decide the proposal.

The first is calibration. In the controlled study, $\beta$ must be near zero for excluded and first-release regions and must rise with duplication count for finalized regions. Otherwise the test is invalid.

The second is the positive control. The language-model forecaster should show clearly positive $\beta$ on series where Recall Residualisation finds heavy leakage. The third is the negative control: $\beta$ should be near zero on Impermanent streams published after model release.

The fourth is the localization result. Localized detection power must exceed both the global and random-window ablations at a 5% false-positive rate, or we drop the localization claim.

Only if the first three hold do we interpret $\beta$ for Chronos, TimesFM, and MOMENT. There, a nonzero value would mean the "zero recall" verdict reflects the probe used rather than the models. A value of zero, with demonstrated power, would settle the question in the models' favor.