TL;DR
This paper investigates "temporal contamination" in Large Language Models (LLMs) used for economic forecasting—a phenomenon where models "remember" future outcomes they encountered during training rather than inferring them from historical text. While the authors prove that modern models clearly possess this "future memory," they find that this advantage is surprisingly difficult to translate into statistically significant gains in real-time forecasting accuracy due to the inherent noise and small sample sizes of macroeconomic data.
What Problem Does This Paper Address?
LLMs are increasingly used to parse central bank communications, like FOMC minutes, to predict economic shifts. However, most popular models were trained on massive web crawls that include news and data from after the documents they are asked to analyze.
This creates a lookahead bias: a model reading a 2021 FOMC statement might already "know" that inflation spiked in 2022 because that fact was part of its training set. This is a form of test-set contamination that runs through time. The researchers set out to answer two critical questions:
- How much of an LLM's apparent forecasting skill is actually just memory of the future?
- Does this "cheating" memory actually lead to better forecasts in a realistic, real-time setting?
Methods
The authors developed a rigorous auditing pipeline to disentangle reading from remembering.
- Temporal Gating: They compared "leaky" modern models (like Mistral and Qwen) against "chronologically consistent" models (ChronoGPT). As shown in Figure 1, they used a temporal gate to ensure a document from year y was only read by a model whose training ended at y-1.
- Date-only Probes: To isolate pure memory, they asked models to predict outcomes (like whether GDP growth was above 2%) given only a date, with no supporting text.
- Real-time Nowcasting: They built a "bridge-factor" nowcaster that integrates LLM-derived policy signals with hard economic data (from the RTDSM) to estimate current-quarter GDP and CPI. This simulates how a professional forecaster would actually use these models.
- Statistical Rigor: They employed the Coroneo–Iacone test for forecast accuracy and applied Holm’s method to adjust for multiple testing, ensuring that any observed "premiums" weren't just statistical flukes.
Datasets
- FOMC Corpus: A curated set of 429 Federal Open Market Committee statements and minutes spanning from February 2000 to January 2026.
- RTDSM: The Philadelphia Fed’s Real-Time Data Set for Macroeconomists, which provides "vintages" of economic indicators exactly as they were published at specific points in history.
- Model Ladder: The study utilized a "ladder" of ChronoGPT and DatedGPT checkpoints (see Figure 2) alongside modern open-weights models including Qwen2.5 (1.5B to 14B), OLMo-2 (7B, 13B), and Mistral-7B.
Key Insights and Ideas
- The "Chance" Baseline is a Trap: You can't just compare a model against a 0.5 AUC (random guessing). The authors found that even "memory-free" gated models can reach an AUC of 0.64 on date probes simply by learning "era priors"—for example, knowing that the late 90s generally had strong growth.
- Modern Models are "Time Travelers": Modern models significantly outperform gated baselines on date-only probes. Mistral-7B achieved an AUC of 0.83 on GDP probes, whereas the gated baseline sat near 0.48. This confirms they have indeed memorized historical outcomes.
- Membership Inference is Confounded by Time: Popular methods for detecting training data contamination (like Min-K% scores) often fail when documents come from different eras. They tend to sort documents by their "era-style" rather than whether the model actually saw them.
- Memory $\neq$ Resolvable Skill: This is the most surprising finding. Despite having "future" knowledge, the "lookahead premium" (the improvement in forecast accuracy) was tiny. In macroeconomic forecasting, the data is so sparse and noisy that even "cheating" doesn't provide a clear, statistically significant edge over a clean model.
Key Results
- Date Probes: Modern models scored between +0.02 and +0.19 higher AUC than the ChronoGPT-2024 baseline on date-only GDP questions.
- Nowcasting Premium: In the real-time GDP nowcasting task, the "leaky" models showed a premium (reduction in RMSE) ranging from -0.9% to +3.4%.
- Statistical Significance: None of the forecasting premiums survived correction for multiple testing. The study revealed that macroeconomic sample sizes are often too small to detect lookahead bias; their test only had 80% power to detect a premium of ~9-13%, which is far larger than what was observed.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope: The study is limited to one central bank (the Fed) and two primary outcomes (GDP and CPI).
- Scale vs. Date: There is an inherent confound between model scale and training cutoff. Chronologically consistent models are currently much smaller than the largest "leaky" models, making it difficult to perfectly isolate the effect of parameters versus the effect of temporal contamination.
- Future Direction: The authors argue that lookahead audits should move away from measuring forecast accuracy alone. Instead, they should focus on direct memory probes compared against "memory-free" readers on the exact same items.
Summary updated