Do Time-Series Foundation Models Remember the Market?

Time-series foundation models are large forecasters pretrained on hundreds of thousands of series, and they now top financial forecasting leaderboards. The worry is that their pretraining data may already contain the stock, crypto and FX histories they are later tested on, so that a high score reflects memory rather than skill. For language models this has been shown: when a model is asked about a named series on a given date, almost all of its apparent forecasting skill turns out to be recall. Numeric foundation models were cleared in that work, but only because they were never given names or dates to recall. Nobody has tested whether they memorize the numbers themselves. We propose to test this with two complementary measurements. The first is a staggered comparison that uses each model's own training-data cutoff date as a natural experiment. The second is a per-window brittleness test: memorized trajectories should stop "predicting" the future once the input is nudged by tiny, economically meaningless noise.

The stakes are concrete. On FinTSB, a numeric foundation model posts the best portfolio result, an annualized Sharpe ratio of 4.327, which would be extraordinary if it were real. Benchmarks such as FinVerse defend against leakage with a fixed 2020 cutoff, yet many models evaluated there were trained on data gathered after 2020. If even part of this performance is recall, current financial leaderboards rank models partly by how much market history they absorbed.

Core Claim

Contamination in numeric forecasters can be identified without a text interface, by combining two signatures. The first is a model-specific step change in skill at that model's training-data cutoff. Market-regime changes hit all models on the same calendar date, while contamination ends for each model on its own cutoff date. The second is brittleness. A genuine forecaster facing low signal-to-noise returns should degrade smoothly when its input window is perturbed far below the scale of daily volatility. A model replaying a memorized continuation should lose its edge abruptly. What is new is the separation of regime shift from contamination through staggered cutoffs, together with a detector whose false-positive and true-positive rates are calibrated on deliberately contaminated models. Neither existing decontamination method provides this for numeric models.

Relation to Prior Work

The motivating evidence comes from Recall Residualisation. That paper asks frontier language models for exact values of public series, such as Fama–French market excess returns and FRED unemployment and inflation, and regresses their forecast signals on that recall. Apparent correlations as high as 0.93 fall to 0.17 or lower, and LeakShare (the fraction of skill explained by recall) is at least 0.97 in 19 of 20 cases. It also reports that Chronos and TimesFM, fed unlabeled numbers, show no recall advantage and perform like an AR(1) baseline.

This finding constrains us, and we take it head on. That test covered four monthly series and measured recall through named dates. It could not detect memorization keyed on the shape of a numeric window. It also did not cover the daily equity and crypto panels where FinTSB reports its large numeric-model gains. If our tests also find nothing on those panels, that clearance becomes much stronger and is worth publishing. We do not reuse its linear residualization step, which needs a "direct recall" query that numeric models cannot answer.

Divergence Decoding removes look-ahead knowledge from language models at inference time by contrasting small "forget" and "retain" models. It is a mitigation, not a measurement, and it was tested on text about M&A deals and filings. We borrow only one idea from it: cheap auxiliary models trained on clean pre-cutoff data can act as a reference for what a non-contaminated predictor would say.

Three benchmarks supply the evaluation settings and each leaves a specific gap. FinTSB offers regime-stratified daily equity backtests with a 0.1% transaction fee. It does not ask whether its winning foundation model memorized the equity histories. FinVerse offers 60,232 core series across asset classes with a 2020-01-01 cutoff. It does not verify whether models released after that date saw post-cutoff data. Impermanent supplies a live "forecast before the outcome exists" protocol, but only for GitHub activity. We use its protocol as the gold-standard negative control in finance.

Research Program

Detector calibration with known contamination. We take a small open numeric foundation model and continue its pretraining on a mixture into which we inject a held-out set of real daily equity and crypto windows. The windows are drawn from FinTSB and from the CTBench Binance hourly returns. Each window is repeated 0, 1, 4 or 16 times. This gives windows whose membership in training is known exactly. It also gives a dose-response curve: how much RankIC, hit ratio and fee-adjusted Sharpe can be inflated by a given amount of exposure.

The brittleness score. For a context window $x$ with realized continuation $y$, let $e(x)$ be the model's forecast error. Let $x_\epsilon$ be $x$ with independent noise of standard deviation $\epsilon\sigma$ added to each return, where $\sigma$ is the window's local return volatility and $\epsilon$ is small (0.05 to 0.2). We define

$$B_\epsilon = \frac{\mathbb{E}[e(x_\epsilon)] - e(x)}{\mathbb{E}[e_{\text{ref}}(x_\epsilon)] - e_{\text{ref}}(x)},$$

where $e_{\text{ref}}$ is the error of a reference forecaster trained from scratch on the same asset only before the window. $B_\epsilon$ near 1 means the model reacts to perturbation like an honest model. A large $B_\epsilon$ means its advantage depended on the exact digits. We calibrate thresholds on the injected windows, using live-logged windows as negatives. Because rescaling and resampling are common in financial data, we also apply the score after rescaling and after split-adjustment-style level shifts.

The staggered cutoff design. For each public model we record its training-data cutoff from documentation, treating an unknown cutoff as an interval. We then estimate how its skill changes, relative to from-scratch references, after its own cutoff, with calendar-period fixed effects absorbing regime shifts. A contamination effect appears as a model-specific drop in skill aligned with its own cutoff rather than with the calendar date. The same models also log forecasts prospectively on daily FinVerse-style series during the project, as a clean live test.

Baselines

The naive reference point is the published leaderboard taken at face value. On FinTSB, Chronos-bolt_Base has ARR 0.366 and annualized Sharpe 4.327, TimeMixer has RankIC 0.107, and Ridge has the lowest MSE (0.012) but ARR 0.026 (FinTSB). We rerun these with identical data, the 0.1% fee and TopK strategies. Our decontaminated estimates count as informative if they move a model's rank or Sharpe outside the rerun's bootstrap interval. On FinVerse we found no reported result on Lacuna for any pre-versus-post-cutoff skill gap.

The strongest existing competitor is post-cutoff-only evaluation, as in FinVerse's fixed date and Impermanent's live stream. We run it on exactly the windows used in our design. We beat it if staggered estimates separate contamination from regime effects where a single cutoff cannot. The test case is two models with different cutoffs whose post-2020 gap differs. A second competitor is Recall Residualisation itself, run on the language-model arm with named series. There its reported LeakShare of at least 0.97 on monthly macro series is the reference figure, and we check whether we reproduce it.

A third competitor is near-duplicate deduplication. We match each test window against published pretraining corpora by normalized correlation. This is run only for open models whose corpora are listed.

The key ablation removes each half of our method in turn: staggered estimates without brittleness, and brittleness without staggering. This shows whether the combination is needed. The injected-contamination models give each variant a true-positive rate to be judged against.

Why not just…?

Why not just evaluate only on post-cutoff data? This is the most serious alternative, and we treat it as one. Post-cutoff evaluation is clean, but it confounds contamination with the 2020-onward regime. It also needs trustworthy cutoffs and discards the long pre-cutoff histories on which FinTSB's regime strata are built. The staggered design is precisely the test of whether post-cutoff evaluation suffices. If model-specific steps vanish once calendar effects are absorbed, we will recommend post-cutoff evaluation plus live logging and drop the claim that our estimator is needed.

Why not just apply Recall Residualisation to numeric models? Its first step asks the model what value a named series took on a named date, and a numeric model has no way to answer that question. The paper's own null result for Chronos and TimesFM reflects this interface limit and is not proof of cleanliness. Our injected-contamination runs show directly whether a contaminated numeric model passes a date-free version of that test while failing ours.

Why not just deduplicate test windows against the training corpus? For open models with published corpora this may work about as well as our method. Our injection experiment will tell us, by measuring deduplication recall on rescaled and resampled copies. It cannot be applied to models whose corpora are undisclosed, which is the case the contamination literature flags as hardest. That is where brittleness and staggering add value.

Decisive Checks

The project succeeds if three things hold. The brittleness detector separates injected windows from live-logged windows well above chance at single-exposure dose. Injected contamination produces a measurable inflation of fee-adjusted Sharpe and RankIC, which establishes that the threat is economically real. And on the public models the staggered estimate is either significantly non-zero for the FinTSB and FinVerse leaders, or tightly bounded near zero. Either outcome changes how those leaderboards should be read. The finding is that numeric models memorize market trajectories if a public model shows high $B_\epsilon$ on pre-cutoff windows and $B_\epsilon$ near 1 on its live forecasts. Uniformly low brittleness together with no cutoff-aligned step would extend the Recall Residualisation clearance to daily financial panels with a properly calibrated test.