When a language model is tested on "future" data, two different things can go wrong, and current evaluations cannot tell them apart. The first is ordinary drift. The world changed, the labels changed, and a classifier trained on last year's examples no longer fits. The second is hidden look-ahead. The pretrained encoder underneath the classifier may have already read text from the test period during pretraining. If so, a benchmark that looks chronological is quietly leaking the future into the representations. This proposal measures both effects at once.
The tool comes from neuroscience, where it is called temporal generalization analysis. You train a simple decoder on brain or network states recorded at one time, freeze it, and test it at every other time. The result is a train-time by test-time matrix of accuracies. We add a third axis to that matrix: the date after which the encoder saw no pretraining text, which we call its cutoff. Quarterly checkpoints with known cutoffs already exist (TimeLMs), so we can ask directly how much of a model's apparent robustness to time comes from having seen the future. Comparing cells of this three-way grid splits each forward performance drop into two parts. One part would be repaired by updating the encoder. The other part only new labels can repair. We then test whether that split predicts which fix actually helps on a given task.
Core Claim
Let $A_\ell(i, j; c)$ be the accuracy of a linear probe defined as follows. It is trained on layer-$\ell$ representations of examples from period $i$ and tested on examples from period $j$. The representations come from an encoder whose pretraining cutoff is $c$.
For a forward gap ($j > i$), the deployment-faithful cell is $A(i,j;c{=}i)$: both the probe and the encoder know only the past. The oracle cell is $A(j,j;c{=}j)$. The total gap between them splits into two terms:
$$ \underbrace{A(i,j;c{=}j) - A(i,j;c{=}i)}_{\text{encoder term } E_{ij}} \;+\; \underbrace{A(j,j;c{=}j) - A(i,j;c{=}j)}_{\text{label term } P_{ij}} $$
The encoder term $E_{ij}$ measures two things at once. It is what updating the encoder alone buys. It is also exactly the inflation a benchmark suffers when it uses an encoder whose cutoff lies past the test period. The label term $P_{ij}$ is what new target-period labels buy once the encoder is current.
Because the split depends on order, we report both orderings and their average.
We make three claims:
- The relative size of $E$ and $P$ varies sharply across tasks, across linguistic levels (entities, facts, task labels), and across layers.
- $E$ predicts the gain from unlabeled continued pretraining on target-period text. This is known as domain-adaptive pretraining, or DAPT.
- $P$ predicts the gain from a few target-period labels.
Together these would explain a known puzzle. Time Waits for No One! found DAPT gives small gains while a little labeled data helps a lot. If our split is right, that result reflects tasks dominated by label drift, such as political affiliation. It is not a general law.
Relation to Prior Work
Luu et al. (Time Waits for No One!) motivate this work and are its closest neighbor. They built year-by-year performance matrices over eight tasks and summarized them with a Temporal Degradation (TD) score, the average drop in score per year of gap between training and test periods. They also showed degradation runs in both directions and is often driven by label shift. However, they used a single fixed encoder and looked only at final-task scores. We vary the encoder cutoff and probe inside the network, and that is what makes the split possible.
TimeLMs supplies the main instrument: RoBERTa models continually pretrained on quarterly Twitter slices. Its authors report roughly a 10% drop after one year of misalignment, but never pair probes with encoders of different cutoffs.
DynamicTempLAMA supplies time-indexed facts split into UNCHANGED, UPDATED, NEW and DELETED. Its authors found that models peak on UPDATED facts at their own timestep. They probe only output predictions, not intermediate layers.
The method is borrowed from cross-temporal decoding. In the neuroscience version (Two Distinct Dynamic Modes Subtend the Detection of Unexpected Sounds), a narrow diagonal of high accuracy marks a passing state and a broad square marks a stable one. Under the Hood carried diagnostic probes into language models, but across token positions within a sentence rather than across calendar time. We reuse the matrix construction at the scale of calendar quarters and read its shape layer by layer.
The leakage literature motivates the third axis. A Critical Study on Data Leakage in Recommender System Offline Evaluation showed future data can shift results by up to 89.5% and flip model rankings. Notably, this happens even when each user's own history is split by time, because leakage travels through shared structure. Pretrained encoders are an analogous shared structure. Leakage flowing through them has been noted for NER on historical tweets (Mitigating Temporal-Drift) but never quantified.
Methods that try to fix drift are not competitors to a diagnostic. Instead they become the interventions whose gains we try to predict. These include factorized temporal adapters (MaT-LoRA) and auxiliary time prediction (Temporal Generalization for Spoken Language Understanding).
Research Program
The core study uses TimeLMs checkpoints from 2019-Q4 to 2022-Q2 as the cutoff axis.
Factual level. We use DynamicTempLAMA facts, building probes on each layer's representation of the masked subject-relation context.
Task level. We build per-quarter Twitter tasks whose labels can be regenerated from timestamped tweets without annotation. Emoji prediction is the clearest case: in TweetEval its labels are derived from the emojis in the tweets themselves. We also build analogous hashtag-defined labels.
Entity level. We hand-annotate a small set of quarterly entity-typing examples. This is the one place the program needs fresh labels.
Collecting timestamped 2019–2022 tweets is a data risk we must manage ourselves; the file does not establish a released corpus. Within this window we fill every cell $(i, j, c)$ and every layer $\ell$ with linear probes that are fit with a fixed tuning budget.
A secondary study extends the analysis to older benchmarks. We pretrain a ladder of small encoders from scratch on cumulative yearly tweet slices, each frozen at year $c$. We then evaluate them on the 2014–2019 NER data of Mitigating Temporal-Drift and on the Twitter tasks of Luu et al. This lets us estimate how inflated published "historical" numbers are when they rest on encoders pretrained past the test year. This study is costlier and is gated on obtaining the corpus.
Beyond the split, we read the shape of each layer's matrix. We expect two patterns:
- Entity and UPDATED-fact probes should show diagonal (passing) structure in upper layers.
- UNCHANGED facts and stable labels should show square (persistent) structure.
We also measure the asymmetry between forward and backward generalization, which Luu et al. observed but did not localize to any part of the network.
Baselines
Naive reference points. The first is a probe evaluated under a random split that mixes all quarters. This shows the leaky number practitioners would otherwise report. The second is the frozen 2019-Q4 encoder with a 2019-Q4 probe applied to all later quarters, which is the do-nothing deployment.
Strongest existing descriptive baseline. This is the TD score and full-task heatmap of Luu et al., computed on the same tasks with a single latest-cutoff encoder. For calibration, Luu et al. report TD = 7.72 on political affiliation and TD = 0.26 on Yelp rating classification. Our split must explain variation that TD cannot; TD alone says nothing about which fix to apply.
Strongest predictive baselines. Two unsupervised drift signals compete with $E$ and $P$ to predict intervention gains:
- the change in TimeLMs pseudo-perplexity on target-period text;
- representation similarity between periods (centered kernel alignment).
No reported result was found in the file for how well either one predicts DAPT or few-shot gains.
Fair comparison. All predictors see the same quarters and the same encoders. They are scored by rank correlation with measured gains from two interventions:
- DAPT on the target quarter;
- fine-tuning on 50–200 target-period labeled examples, a budget chosen to match the setting where Mitigating Temporal-Drift reports 2.70–5.71% F1 gains from trend-based selection.
We beat the baselines if $E$ correlates with DAPT gains, and $P$ with label gains, significantly better than perplexity or similarity do across tasks and quarters.
Ablations. The key ablation collapses the cutoff axis to a single latest encoder. This yields Luu-style matrices with probes; if prediction does not worsen, the third axis is not doing work. A second ablation probes only the last layer, to test whether layer resolution matters.
Why not just…?
Why not just read Luu-style end-task heatmaps after fine-tuning, instead of linear probes? Fine-tuned performance is what users care about, and probes are only a proxy. This is a real threat. We fine-tune on a subset of $(i,j,c)$ cells and check that the split computed from fine-tuned scores agrees with the probe-based one. If they disagree, the probe matrices are not a valid cheap substitute, and we report that. Probes earn their place only because they make the full three-axis grid affordable.
Why not just measure drift without labels, via perplexity or representation similarity between periods? These signals see changes in text but not changes in the mapping from text to label. Luu et al. show that label shift drives much of the degradation on tasks like political affiliation. We therefore predict that unsupervised signals will track $E$ but miss $P$. The head-to-head prediction experiment in the Baselines section tests this directly, and if perplexity predicts both kinds of gain equally well, our split adds nothing.
Why not just enforce strict chronological discipline and only use encoders with clean cutoffs? For new benchmarks that is correct, and the deployment-faithful cell is exactly that. But it gives one number, not a diagnosis of what to update. For historical benchmarks evaluated with modern encoders, a clean-cutoff encoder often does not exist. Measuring $E$ tells readers how much existing results should be discounted.
Why not just run DAPT, since it is cheap? Luu et al. found its gains small on their tasks. We do not dispute that finding. We predict DAPT helps only where $E$ is large, plausibly entity and fact strata, and the program tests whether that prediction holds.
Proposed Experiments
Inflation check. On the small-encoder ladder, we report $E$ for the 2014-trained probe tested on 2019 NER data, comparing an encoder with cutoff 2014 against one with cutoff 2019. A non-trivial $E$ quantifies pretraining look-ahead in a published-style setup. An $E$ near zero would itself be a useful negative result.
Decomposition check. Across DynamicTempLAMA strata, we expect $E$ to be large for NEW and UPDATED facts and near zero for UNCHANGED facts. This follows from the authors' observation that unchanged facts are retained across checkpoints.
Decisive predictive test. Over all task-quarter pairs, $E$ should rank-correlate with measured DAPT gains and $P$ with few-shot label gains, beating perplexity and similarity predictors and the collapsed-cutoff ablation. If $E$ and $P$ fail to separate the two interventions, the central claim is wrong. In that case the contribution shrinks to a descriptive layer-wise atlas of drift.
References
- TimeLMs: Diachronic Language Models from Twitter
- Time Waits for No One! Analysis and Challenges of Temporal Misalignment
- Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views
- Two Distinct Dynamic Modes Subtend the Detection of Unexpected Sounds
- Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
- A Critical Study on Data Leakage in Recommender System Offline Evaluation
- Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp
- Manifold-Aware Temporal Domain Generalization for Large Language Models
- Temporal Generalization for Spoken Language Understanding
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification