Reading or Remembering? Measuring Temporal Contamination in Language Models

Arunkumar V, Gangadharan G. R., S. SenthilkumararXiv 2026 · arXiv

TL;DR

This paper investigates "temporal contamination" in Large Language Models (LLMs) used for economic forecasting—a phenomenon where models "remember" future outcomes they encountered during training rather than inferring them from historical text. While the authors prove that modern models clearly possess this "future memory," they find that this advantage is surprisingly difficult to translate into statistically significant gains in real-time forecasting accuracy due to the inherent noise and small sample sizes of macroeconomic data.

What Problem Does This Paper Address?

LLMs are increasingly used to parse central bank communications, like FOMC minutes, to predict economic shifts. However, most popular models were trained on massive web crawls that include news and data from after the documents they are asked to analyze.

This creates a lookahead bias: a model reading a 2021 FOMC statement might already "know" that inflation spiked in 2022 because that fact was part of its training set. This is a form of test-set contamination that runs through time. The researchers set out to answer two critical questions:

  1. How much of an LLM's apparent forecasting skill is actually just memory of the future?
  2. Does this "cheating" memory actually lead to better forecasts in a realistic, real-time setting?

Methods

The authors developed a rigorous auditing pipeline to disentangle reading from remembering.

  • Temporal Gating: They compared "leaky" modern models (like Mistral and Qwen) against "chronologically consistent" models (ChronoGPT). As shown in Figure 1, they used a temporal gate to ensure a document from year y was only read by a model whose training ended at y-1.
  • Date-only Probes: To isolate pure memory, they asked models to predict outcomes (like whether GDP growth was above 2%) given only a date, with no supporting text.
  • Real-time Nowcasting: They built a "bridge-factor" nowcaster that integrates LLM-derived policy signals with hard economic data (from the RTDSM) to estimate current-quarter GDP and CPI. This simulates how a professional forecaster would actually use these models.
  • Statistical Rigor: They employed the Coroneo–Iacone test for forecast accuracy and applied Holm’s method to adjust for multiple testing, ensuring that any observed "premiums" weren't just statistical flukes.
Figure 1: The pipeline at one forecast origin τ. A temporal gate routes each FOMC document to the ChronoGPT checkpoint whose training ended the year before it; ungated readers see every document. An output gate turns next-token label scores into a five-dimensional policy signal and abstains when three wordings of a question dis- agree in sign. The same readers answer date-only memory probes. At each origin the nowcaster takes one vintage of the Philadelphia Fed Real-Time Data Set (RTDSM), the eleven indicators as published by that date. It fills the unpublished months, the ragged edge, with a second-order autoregression on each indicator’s monthly changes (AR(2) fill) and reduces the target’s block of indicators to three principal components (block PCA). A ridge re- gression of the target on these factors and any text signal, the ridge bridge, gives the nowcast ˆyτ. Every nowcast is scored against the first release, the first published estimate of the quarter.

Datasets

  • FOMC Corpus: A curated set of 429 Federal Open Market Committee statements and minutes spanning from February 2000 to January 2026.
  • RTDSM: The Philadelphia Fed’s Real-Time Data Set for Macroeconomists, which provides "vintages" of economic indicators exactly as they were published at specific points in history.
  • Model Ladder: The study utilized a "ladder" of ChronoGPT and DatedGPT checkpoints (see Figure 2) alongside modern open-weights models including Qwen2.5 (1.5B to 14B), OLMo-2 (7B, 13B), and Mistral-7B.
Figure 2: The study on one calendar. ChronoGPT checkpoints end on 31 December of each year from 1999 to 2024; a document is read by the checkpoint that ends the year before it. The modern readers were released in 2024, so their training text covers most of the test window. Four fixed ChronoGPT checkpoints, and three DatedGPT checkpoints on the probes, form the ungated ladder. Specification selection uses only origins before the test window.

Key Insights and Ideas

  • The "Chance" Baseline is a Trap: You can't just compare a model against a 0.5 AUC (random guessing). The authors found that even "memory-free" gated models can reach an AUC of 0.64 on date probes simply by learning "era priors"—for example, knowing that the late 90s generally had strong growth.
  • Modern Models are "Time Travelers": Modern models significantly outperform gated baselines on date-only probes. Mistral-7B achieved an AUC of 0.83 on GDP probes, whereas the gated baseline sat near 0.48. This confirms they have indeed memorized historical outcomes.
  • Membership Inference is Confounded by Time: Popular methods for detecting training data contamination (like Min-K% scores) often fail when documents come from different eras. They tend to sort documents by their "era-style" rather than whether the model actually saw them.
  • Memory $\neq$ Resolvable Skill: This is the most surprising finding. Despite having "future" knowledge, the "lookahead premium" (the improvement in forecast accuracy) was tiny. In macroeconomic forecasting, the data is so sparse and noisy that even "cheating" doesn't provide a clear, statistically significant edge over a clean model.

Key Results

  • Date Probes: Modern models scored between +0.02 and +0.19 higher AUC than the ChronoGPT-2024 baseline on date-only GDP questions.
  • Nowcasting Premium: In the real-time GDP nowcasting task, the "leaky" models showed a premium (reduction in RMSE) ranging from -0.9% to +3.4%.
  • Statistical Significance: None of the forecasting premiums survived correction for multiple testing. The study revealed that macroeconomic sample sizes are often too small to detect lookahead bias; their test only had 80% power to detect a premium of ~9-13%, which is far larger than what was observed.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study is limited to one central bank (the Fed) and two primary outcomes (GDP and CPI).
  • Scale vs. Date: There is an inherent confound between model scale and training cutoff. Chronologically consistent models are currently much smaller than the largest "leaky" models, making it difficult to perfectly isolate the effect of parameters versus the effect of temporal contamination.
  • Future Direction: The authors argue that lookahead audits should move away from measuring forecast accuracy alone. Instead, they should focus on direct memory probes compared against "memory-free" readers on the exact same items.

Summary updated

Abstract

Language models are now used to read central-bank text for economic forecasting, but most were trained on text written after the documents they read. A model reading a 2021 FOMC statement may already know that inflation rose in 2022, so part of its apparent skill can be memory of the outcome, a temporal form of test-set contamination. We measure this memory on FOMC statements and minutes from 2000 to 2026 by comparing modern open models with chronologically consistent models, which are trained only on text from before a fixed date. Asked about past GDP outcomes given only a date, models that cannot know the answer still reach an AUC of up to 0.64 from priors on dates and eras, so chance is the wrong baseline. Against these memory-free models on the same questions, most modern models recall which years had strong growth. Membership-inference scores sort documents by era and document type rather than by whether a model saw them, as exact counts in an open pretraining corpus show. In real-time nowcasts, every lookahead premium we estimate on US GDP is smaller than the forecast comparison can detect at macroeconomic sample sizes, and none on GDP or inflation survives correction for multiple testing. Lookahead audits should therefore measure memory directly against memory-free models on the same items.
BibTeX
@misc{v2026reading,
  author = {Arunkumar V and Gangadharan G. R. and S. Senthilkumar},
  title = {{Reading or Remembering? Measuring Temporal Contamination in Language Models}},
  year = {2026},
  eprint = {2611.21554},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21554},
}
Figure