Detecting Memorized Benchmarks in Time-Series Foundation Models by Sliding the Window

Time-series foundation models are large forecasters pretrained on huge collections of public numeric series, used zero-shot on new data. They are scored on public benchmarks that were almost certainly part of what they were trained on. For text models there are many tools to detect this contamination, meaning test data that leaked into training. For raw numeric streams there are essentially none. Every existing detector works on discrete text: it swaps synonyms, asks multiple-choice questions, or prompts for a value at a named date. Our proposal uses the operation these models are built around, cutting a long recording into overlapping windows, as the probe itself. A model that has memorized a benchmark series behaves differently from a model that forecasts it in three ways. It needs very little context to be accurate. Its error barely grows as it predicts further ahead. Its accuracy collapses abruptly when the input is nudged by small amounts of noise. We turn these three signatures into a window-level contamination score that needs only forward passes on a frozen model.

The same testbed answers a question that sliding-window augmentation research has never asked. When pretraining data is cut into windows with heavy overlap, a common way to multiply scarce recordings, does the overlap teach the model to ignore when a pattern starts, or does it make the model memorize specific trajectories? We plant benchmark series into the pretraining of small forecasters at controlled window strides and repetition counts. This lets us measure how the augmentation choice governs memorization, and it gives us ground-truth labels to validate the detector.

Core Claim

Genuine forecasting and memorized recall of a numeric series leave distinguishable fingerprints when context length, window offset and input noise are varied. We call this set of controlled variations the window-response surface. We claim three things. First, a detector built on this surface can flag contaminated series in a frozen time-series foundation model without access to its training data or text metadata. Second, its flags will predict the real-world generalization gap between old public benchmarks and post-cutoff data. Third, the stride overlap used during pretraining is a significant driver of how deeply benchmark series are memorized.

Concretely, for a series segment we measure forecast error $e(L, h, \sigma)$. Here $L$ is the context length fed to the model, $h$ is the forecast horizon, and $\sigma$ is the standard deviation of Gaussian noise added to the context, relative to the series scale. Each measurement is averaged over many start offsets. We compare this surface to the same surface computed on reference series: phase-randomized surrogates, which keep the original's power spectrum and autocorrelation but scramble the exact trajectory, and post-cutoff segments from the same source where they exist. The contamination score combines three quantities. The first is the context gap, how much better the original does than its references at short $L$. The second is the horizon slope, the growth of error with $h$ relative to the references. The third is the sharpest single-step rise in error as $\sigma$ increases.

Where This Sits

The work is motivated by two papers that found the gap. Evaluating Time Series Foundation Models for Electricity Price Forecasting compared models on the public GEFCom2014-P benchmark and on a new post-cutoff GridStatus2025 set. TOTO 1.0 did very well on the old data and crashed on the new. That paper diagnosed contamination only at the whole-dataset level, and only because a clean post-cutoff dataset happened to exist. Recall Residualisation prompts language models for values at named dates and regresses that recall out of their apparent skill. It found that nearly all of the skill was recall, and that dedicated numeric forecasters fed raw series without dates performed only at the level of a first-order autoregressive model. The two papers are in tension: one says numeric interfaces do not leak, the other shows a numeric model that apparently memorized a benchmark. Our detector is the tool that can settle which series, in which models, are actually memorized. It works exactly where recall residualization cannot, on series with no textual date to prompt for.

The method borrows its logic from text-side detectors. CoDeC observes that adding in-context examples raises confidence on unseen data but lowers it on memorized data, because memorization sits in narrow, brittle optima. Our context-length axis is the time-series analogue of that observation. PEARL flags memorized strings by the sharpest drop in fidelity across increasing bit-flip perturbations. Our noise axis adapts that idea to real-valued input. CDD detects memorization through collapsed diversity across sampled outputs. We use it as a baseline on probabilistic forecasters rather than as the core signal, because it is expensive and blind to deterministic point forecasters.

Two further lines of work constrain the design. Test of Time shows that paraphrasing launders memorized text past cutoff-based detection. This warns us that shifting the window offset may similarly launder numeric memorization, so offset-shifted benchmarks are an explicit stress test rather than an assumed-clean control. Unraveling the Complexity of Splitting Sequential Data defines temporal leakage, where near-identical adjacent windows fall on both sides of a split. That paper stays within classical models and train/test split design. We take its concern to the pretraining corpus of foundation models and to after-the-fact auditing.

On the augmentation side, Luo et al. cut overlapping crops from EEG trials (the sliding window cropping strategy) to learn features that persist across time. Ghiasi et al. randomly crop sleep recordings to make arousal detection insensitive to shifts. Both treat overlap as purely beneficial. Neither asks what it does to memorization.

Research Program

The first component is a planted-contamination testbed. We pretrain a family of small patch-based transformer forecasters on a public mixture of series. Into that mixture we insert chosen benchmark series (GEFCom2014-P prices, BCI Competition IV 2a EEG channels, PhysioNet 2018 polysomnography channels, and the macroeconomic series used by Recall Residualisation) under three controlled variables. The first is window overlap ratio $(W-S)/W$, where $W$ is window length and $S$ is stride, ranging from 0 to 0.95. The second is the number of repetitions of each series. The third is the offset policy, either fixed-grid windows or random-offset crops. Held-out series from the same sources serve as clean negatives. This produces exact contamination labels per series and per window.

The second component is the detector. For each test segment we compute the window-response surface on a small grid of $L$, $h$ and $\sigma$, averaged over start offsets. We generate matched phase-randomized surrogates, and we fit a small logistic combiner of the three statistics on planted-model labels. The frozen combiner is then transferred unchanged to public models. The cost is a few hundred forward passes per series and no gradients.

The third component is the augmentation study. Using the same planted models, we measure memorization depth: how much the error on planted series falls below that of their surrogates. We measure it as a function of overlap ratio, repetition count and offset policy. We also measure the effect of offset-shifted evaluation on detectability, which tests whether random-offset cropping launders memorization or genuinely removes it.

The fourth component is a real-model audit. We apply the frozen detector to Chronos, TimesFM and TOTO 1.0 on GEFCom2014-P, the macro series, and the Impermanent GitHub streams. Where post-cutoff data exist (GridStatus2025, and the live windows of Impermanent), we test whether the flags predict the old-to-new performance gap.

Baselines

The naive reference point is to take the benchmark score at face value and compare the foundation model to SeasonalNaive and a first-order autoregressive model on the same windows, treating a large margin as suspicious. Recall Residualisation reports that dedicated time-series models on raw numeric macro series sat at parity with the first-order autoregressive baseline, and Impermanent reports SeasonalNaive as competitive in MASE (mean absolute scaled error), so this margin is a meaningful contamination proxy. We compute it on identical windows and horizons. To beat it, our detector must achieve higher area under the ROC curve (AUC) on planted labels at fixed false-positive rates.

The strongest existing temporal method is the post-cutoff comparison of Pan and Ezzat. Their best GridStatus2025 system, an ensemble of Chronos 2 and CING-LEAR, reports 3.922 MAE (mean absolute error) and 1.541 aQL (average quantile loss). For TOTO's GEFCom-versus-GridStatus gap, and for any window-level contamination detector on numeric foundation models, no reported result was found on Lacuna. We run their two-dataset protocol verbatim as a dataset-level oracle. Our detector beats it if it reaches the same verdict on each model without the post-cutoff data, and additionally ranks individual series within GEFCom2014-P.

The strongest text-derived detectors, adapted to numbers, are the second competitor family. We implement a CDD-style output-diversity score on the sampled forecasts of Chronos, and a CoDeC-style score that prepends unrelated series windows as "in-context" material and measures the change in likelihood. Both get the same forward-pass budget as our method and the same labeled planted models for threshold tuning.

Ablations isolate each ingredient: each of the three axes alone, surrogate references alone (the "surrogate gap"), and the full score with post-cutoff references swapped in for surrogates.

Why not just…?

Why not just evaluate on post-cutoff data? When clean future data exist, this is the ground truth, and we use it as such. But it only says that a model did worse on new data, which also happens under ordinary distribution shift. Test of Time shows that cutoff decay can vanish under transformation even when memorization persists. And for small clinical or lab recordings such as EEG and polysomnography, no post-cutoff cohort exists. The audit tests whether our score recovers the post-cutoff verdict where it is available, which is what would justify using the score where it is not.

Why not just compare to phase-randomized surrogates and stop? This is the most serious threat, because it is simple and may carry most of the signal. Its weakness is predictable. A genuine forecaster that exploits nonlinear structure also beats linear-structure surrogates, so real skill and recall would look alike. The noise-sensitivity axis is meant to break that tie: real nonlinear skill should degrade smoothly with noise, while recall should fall off a cliff. If the surrogate gap alone matches the full score on planted labels and on the GridStatus check, we will report the simpler test as the method.

Why not just search the pretraining corpus for near-duplicate windows? For models with open corpora this is exact, and in the planted testbed it is our label source. It fails for models whose training mixtures are not fully public. It is also blind to laundered copies, such as rescaled, resampled or offset versions. A detector that works from behavior alone is the only option in those cases, and the laundering stress tests measure how much is lost compared with lookup.

Proposed Experiments and Decisive Checks

The first decisive check is on planted models. The full score must separate planted from held-out series at clearly higher AUC than the naive margin, the CDD-style score and the CoDeC-style score. It must hold up under offset-shifted, resampled and noise-rescaled copies of planted series. A detector that only catches fixed-grid copies fails this check.

The second is the augmentation result. We expect memorization depth to rise with overlap ratio and repetitions under fixed-grid windowing. We expect random-offset cropping to reduce it only partly. A flat curve would be a clean negative result, and it would tell practitioners that heavy overlap is safe.

The third is real-world transfer. TOTO 1.0 on GEFCom2014-P should score as contaminated, and Chronos and TimesFM on raw macro series should score near clean, in line with their parity with the autoregressive baseline. Across the audited models, the per-model score should correlate with the GEFCom2014-to-GridStatus2025 gap and with performance drops on Impermanent's live windows. If the score fails to predict these gaps, the claim that behavior alone can audit numeric foundation models is falsified.