A language model can find a test document easy for two different reasons. It may have read that exact document during training, which is contamination. Or the document may simply be written in language the model knows well. Today's detectors mostly read raw predictability (low loss, low perplexity), so they cannot separate these cases. Comparing performance before and after a model's training cutoff does not separate them either, because language itself changes over time. We propose a probe built on long context windows. Take a long document, such as a full arXiv paper or a book. Ask the model to predict tokens that have not yet appeared anywhere in the document. Run this twice: once with the document's own earlier sections in the distant context, and once with an equally long, same-month, same-topic stranger document in that slot. In both runs the model sees the same immediate local text before the target. If the model has never seen the document, its own distant past should not help it guess content the document has not yet stated, beyond what a topical stranger would supply. If it has memorized the document, the distant past tells the model which document it is reading, and recall of the unstated content follows.
The probe is designed so that language change cancels out. Both conditions carry the same era, style and topic, so drift in vocabulary or the post-2023 surge of LLM-assisted writing affects both equally. What remains is document-specific knowledge stored in the weights. The probe also asks a question that prior work explicitly left open: whether long prompts re-expose memorization that paraphrasing hides in short prompts.
Where the gap is
Three findings motivate this work. Test of Time shows that the "performance drops after the cutoff" signal is unreliable. On verbatim cloze questions from 20,277 math and physics arXiv papers, GPT-4o-mini drops 3.83 points after its cutoff. On LLM-rewritten problems built from the same papers, the drop disappears, and post-cutoff physics is 2.19 points better. Influence-function tracing on OLMo2-7B-Instruct finds the source paper 77.5% of the time for verbatim items but only 17.5% after rewriting. All of this was measured on short snippets. Is Your LLM Outdated? (FreshBench) adds a confound in the opposite direction. arXiv text written after March 2024 becomes more predictable, apparently because of LLM-assisted writing. A loss-based detector would wrongly read that as contamination. CoDeC supplies the closest method. It prepends in-context examples from the same dataset and observes that confidence rises for unseen data but falls for memorized data. It reaches 99.9% dataset-level AUC across 40 open-weight models, yet its own analysis reports ambiguous scores between 60% and 80% when partial contamination and nearby domain shift mix. CoDeC prepends other examples, and its contexts are short.
Our probe differs from CoDeC in three ways. The context we vary is the document's own distant history, set against a matched stranger. We score only first-occurrence content, meaning tokens that cannot be copied from earlier context. And we operate at the document level and at 32k–128k lengths, where a model can be cued by an entire paper. CoDeC's "stranger prefix" condition is essentially our control arm.
The long-context architecture literature acts as a constraint on the probe. Sparse attention and KV-cache retrieval decide whether the distant past is actually read. InfLLM-V2 switches between dense and sparse attention using the same weights, and it retains 98.1% of dense performance on RULER at 32k. Native Sparse Attention and ParisKV likewise discard or retrieve distant blocks. A probe that depends on far context must be checked under these mechanisms. That check is also a chance to ask whether block selection behaves differently for memorized documents.
Core Claim
For a long document $d$, pick target spans and keep the same local window of $w$ tokens (for example 1,024) immediately before each target in both conditions. The far prefix beyond that window is either $d$'s own earlier text or an equal-length stranger $s$, matched on arXiv category, publication month and length. Let $\mathcal{N}(d)$ be the target tokens whose surrounding 3-gram never appears in $d$'s own prefix. These are content the document has not yet stated. The statistic is
$$\Delta_{\text{novel}}(d) = \frac{1}{|\mathcal{N}(d)|}\sum_{t \in \mathcal{N}(d)} \big[\ell_t(\text{stranger far prefix}) - \ell_t(\text{own far prefix})\big],$$
where $\ell_t$ is the model's per-token loss. For unseen documents we predict $\Delta_{\text{novel}}$ near zero, whether or not they are drifted, because the own history adds only topical help that the stranger also gives. For memorized documents, verbatim or paraphrased, we predict it is clearly positive. Drift and the LLM-writing effect enter both terms equally and should cancel.
There is a competing hypothesis, and the controlled experiment below resolves it. The local window alone may already trigger full recall, so the distant own history adds nothing. In that case $\Delta_{\text{novel}}$ would be flat in both cases, and the probe would fail.
Research Program
The first stage is a controlled injection study, where membership is known exactly. We take a small open-data model and continue pretraining it on a mixture that includes post-cutoff arXiv papers from the RealMath collection. Each injected paper appears in one of three forms: verbatim, LLM-paraphrased in the style of Test of Time's rewrites, or not at all. Exposure counts are 1, 2 or 8. We then measure $\Delta_{\text{novel}}$ as a function of far-prefix length, from 0 to 32k tokens or the longest window the model supports. This stage tells us whether the signal exists, which direction it takes, and how it scales with exposure. The detector threshold is calibrated here and is never retuned on downstream models.
The second stage transfers the probe to open-data models whose pretraining corpus is public, such as the OLMo2 family over Dolma. There, membership of arXiv papers and Project Gutenberg books can be checked by corpus lookup. We also apply it to long-window open-weight models with known cutoffs, where membership is known only through the temporal split. The temporal split serves as a noisy label, not as the detector.
The third stage is the architectural check. We run InfLLM-V2's dense and sparse modes on identical weights and ask two questions. Does $\Delta_{\text{novel}}$ survive sparse block selection? Do the blocks selected from the own prefix concentrate on fewer, more specific regions for memorized documents? A positive answer to the second question would give a cheaper signal, read off the attention selections.
Baselines
The naive reference point is the temporal split itself. We compare a model's loss or accuracy on pre-cutoff versus post-cutoff documents and call a slice contaminated if the pre-cutoff side is easier. Test of Time's +2.19-point reversal and FreshBench's post-March-2024 loss drop show that this reference point can point the wrong way. We report it on identical documents to make that failure visible.
Loss-only detectors are the next tier. These are perplexity, Min-K%, and the Zlib-compression ratio, which CoDeC uses as comparison points. We run them on the same target tokens, with the same models and the same local window, so the only difference from our probe is the absence of the own-versus-stranger contrast. No reported document-level numbers for these on long arXiv texts were found on Lacuna.
The strongest competitor is CoDeC, which reports 99.9% dataset-level AUC separating seen from unseen benchmarks. We run it in its original few-shot form. We also run a stretched form in which the prepended examples fill the same far-prefix budget as our probe, using other papers from the same month and category. That stretched form is our stranger arm scored the CoDeC way, so the comparison isolates the own-history contrast itself and not context length or compute. Both methods get the same models, token budget, and calibration data from the injection study. To beat CoDeC, our probe must exceed its document-level AUC on paraphrased injected papers and on the post-March-2024 slice, which are the two regimes where CoDeC's scores are expected to land in its ambiguous band. It must also not lose to CoDeC on verbatim contamination. Matching CoDeC at the dataset level is not enough, because that metric is already saturated.
A costly reference is EK-FAC influence tracing as in Test of Time, with its reported 77.5% verbatim and 17.5% paraphrased hit rates on OLMo2-7B-Instruct. We run it on a subsample to show what an expensive white-box method recovers on the same documents.
Two ablations isolate the key ingredient. The first scores all target tokens instead of first-occurrence tokens only. We expect this to reintroduce copying effects that occur for unseen documents too. The second removes the stranger arm, comparing own far prefix against no far prefix. We expect this to reintroduce drift, because any long context helps on unfamiliar language.
Why not just…?
Why not just use the pre/post-cutoff accuracy gap? It measures time, not memorization. Test of Time shows it vanishes or reverses under paraphrase. FreshBench shows LLM-assisted writing makes recent text easier, which would mimic contamination. Our probe holds the era fixed inside each comparison, so these confounds enter both sides.
Why not just run CoDeC with longer prompts? This is a real threat, and we include it as the stretched-CoDeC baseline. If prepending more same-domain examples separates paraphrased memorized papers as well as the own-history contrast does, then the own-history idea adds nothing and we will report that. Our reason to expect a gap is mechanistic. Other examples perturb a memorized state in general terms. The document's own history is the only context that specifically identifies which memorized document to recall. The paraphrase condition of the injection study settles the question directly.
Why not just calibrate perplexity against a reference model with a later or earlier cutoff? A reference model shares the drift problem in a different direction. It also fails when it has itself seen the document, which is common for public arXiv and Gutenberg text. Our stranger arm acts as a within-model reference that needs no second model. We still report a reference-ratio baseline using a later-cutoff model of the same family, and we concede that where that model is known to be clean, the baseline may be competitive.
Why not just use influence functions? They need training-data access and heavy compute. Test of Time shows their hit rate falls from 77.5% to 17.5% under paraphrase, exactly the regime we target.
Proposed Experiments
The decisive check comes from the injection study. Plot $\Delta_{\text{novel}}$ against far-prefix length for never-seen, paraphrased, and verbatim papers at each exposure count. The probe succeeds if the never-seen curve stays near zero while both injected curves rise with prefix length. It fails if the curves overlap, or if all of the signal sits in the local window, which is the competing hypothesis. A second check uses the post-March-2024 RealMath slice on open-data models with earlier cutoffs. Those papers are unseen but unusually predictable. Loss-only detectors should flag them, and our probe should not. A third check uses public-domain Gutenberg books, which are certainly in the training data, against post-cutoff long-form text. Pink et al.'s study of order memory already relies on a 1926 novel that such models have surely read. That case gives a natural positive control where $\Delta_{\text{novel}}$ should be large.
To confirm that the models really read far context, we report RULER and LongBench scores alongside the probe. A model whose effective context is short will show a flat probe regardless of membership. RULER's own finding that only about half of models claiming 32k+ tokens perform well at that length makes this control necessary. The architectural experiment reports AUC for dense versus sparse InfLLM-V2 on identical documents, plus the overlap between the selected own-prefix blocks and the source passages for memorized papers. A large AUC drop under sparse mode would be a practical warning that sparse inference can hide contamination from probes that rely on far context.
References
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Is Your LLM Outdated? A Deep Look at Temporal Generalization
- Detecting Data Contamination in LLMs via In-Context Learning
- InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
- Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models