Large language models look increasingly capable on historical text: they date court records, order events in colonial letters, and tag seventeenth-century French. Nobody knows how much of this is skill and how much is recall. Many of the best-known historical collections have been transcribed and posted online, so a model may have read the exact test documents during training. Contemporary benchmarks handle this risk with fresh text, meaning documents written after the model's training cutoff. Historical text cannot be fresh in calendar time. It can be fresh in a different sense: millions of archival pages have never been transcribed or put on the web, and handwriting recognition is only now turning them into text.
We propose to build historical benchmarks from matched pairs of archival documents. Within each pair, the two documents come from the same series and decade, but one has long been on the web and the other has never been published. Labels for both are generated automatically from the archive's own bookkeeping, such as the date line that heads each entry or the regnal year in a court annal, and these cues are then hidden from the text. The difference in a model's accuracy between the published and unpublished halves estimates how much of its historical "competence" is memorization. A small model pre-trained on a corpus we fully control serves as the reference point that separates memorization from simple differences in difficulty between the two halves.
Core Claim
A model that was never shown either half of the benchmark can still score differently on the two halves, for example because published documents tend to be famous and cleanly written. A model's raw published-minus-unpublished gap is therefore not a contamination measure. We define it relative to a reference model:
$$C(m) = \big[A_m(\text{pub}) - A_m(\text{unpub})\big] - \big[A_r(\text{pub}) - A_r(\text{unpub})\big],$$
Here $A_m$ is the accuracy of the evaluated model $m$ on a half, and $r$ is a domain-specific reference model whose pre-training corpus provably excludes both halves. $C(m)$ is a difference-in-differences. It is near zero if $m$ has real historical skill, and positive if $m$ has seen the published documents. The claim we test is that frontier models show large positive $C$ on well-known collections. Domain pre-trained encoders, whose training data is known, should not.
Each ingredient comes from one of the three directions. Distant supervision, meaning labels derived automatically from existing structure rather than from annotators, supplies the labels. Domain-specific historical pre-training supplies the reference model, and it is the only kind of model whose training data can be controlled. Temporal benchmark decay supplies the question being asked.
Relation to Prior Work
Three papers motivate the question by showing that contamination inflates scores on modern text. LatestEval builds reading-comprehension tests from text published in the past week. It shows that models have lower perplexity on older benchmarks such as BoolQ, which suggests those benchmarks were in the training crawl. LiveAoPSBench splits math problems by posting date and finds accuracy drops of 4.02% to 23.6% on post-cutoff problems. When Benchmarks Age shows that up to 63.78% of BoolQ's time-sensitive answers have since become wrong. All three define freshness by calendar date. All three also depend on the live web, which has no way of reaching unindexed archives, a failure mode that UIS-Digger highlights. We replace "written after the cutoff" with "never published," a condition that holds for text of any age.
Two papers supply our method. Effective Distant Supervision for Temporal Relation Extraction labels event pairs using explicit time expressions in modern news and then masks those cues, so the model has to rely on context. We apply the same label-then-mask approach to archival date lines and regnal-year formulas. TDBench generates questions from database time intervals using Allen's interval algebra, the 13 possible relations between two time spans such as "overlaps" or "contains." It reports a 21.7% average drop when models must also give the correct dates. We borrow its interval-question templates for officeholding spans mined from archives. Unlike TDBench, we work from noisy semi-structured records rather than clean tables. Unlike both papers, we use the labels to measure contamination, not to train models or measure factual drift.
Three historical-modeling papers both make the reference model possible and limit how we can build it. HUE already derives a chronological-attribution task from Joseon court records. It also finds that masking named entities raises F1 from 75.74 to 82.45, which suggests that names act as shortcuts. HUE never asks whether its source texts circulate online. Language Models Lack Temporal Generalization and Bigger is Not Better and From FreEM to D'AlemBERT show that small models trained on period text beat much larger contemporary models. This is why a 6GB-scale reference model can be a competent comparison point and not merely a weak one. The first of these papers also reports that models built on the RoBERTa architecture vary noticeably across random seeds. That finding forces us to report $C$ with confidence intervals computed over seeds.
Research Program
We will build matched benchmarks for two collections whose web presence we will audit first. The first is Joseon Hanja court records, the AJD, DRS and DRRI archives from HUE, where we expect parts to be heavily published. The second is the VOC archive of 5 million handwritten pages from the GLOBALISE project, much of which exists only as recent handwriting-recognition output. For each candidate document we establish publication status from open-web search hits and from n-gram overlap with publicly available crawl snapshots. Documents we cannot classify confidently are discarded.
Pairs are matched on series, decade, length and document type. Both halves are re-transcribed from scans with the same handwriting-recognition system, so that differences in recognition errors do not stand in for publication status. Recognition errors are a known hazard for historical text (Klaus et al.).
The task generators read structure the archive already contains. For dating, the label is the entry's date header or regnal year. For ordering, the label comes from comparing the date headers of two entries, as in the temporal-relation paper. For interval questions, appointment and dismissal entries are mined into officeholding spans and turned into Allen-relation questions. Before any model sees the text, the date header is removed and all in-text date expressions, era names and person names are masked. Historians will check a sample of the generated items for faithfulness.
The reference models are a continued-pre-training run in the style of HUE and an encoder in the style of GloBERTise. Their corpora have both benchmark halves removed by exact and fuzzy deduplication. For calibration, a second copy of each reference model is trained with the published half deliberately included. This tells us how large $C$ becomes under known contamination.
Baselines
The naive reference point is the raw published-minus-unpublished gap of each frontier model, computed without the reference-model correction. If this gap matches $C$, the reference model contributes nothing. If the reference model itself shows a large gap, the raw gap mostly measures differences in difficulty between the halves.
The strongest existing method for detecting contamination is the perplexity comparison from LatestEval. We will compute it for each open-weight model, using perplexity on the published half against the unpublished half, on the same documents with the same tokenization. There is no reported result on Lacuna for perplexity-based contamination detection on historical text. On historical text, perplexity is high for everyone because of archaic spelling. Our measure beats this baseline if, on the calibration models where the truth is known, $C$ separates contaminated from clean models with higher accuracy than the perplexity gap does.
The strongest task performer is HUE's continued-pre-training model. The best number reported for the closest existing task, chronological attribution, is an F1 of 79.33 for AnchiBERT plus AJD/DRS, against 54.26 for a BERT baseline that received no historical pre-training (HUE). We will retrain it under our deduplicated corpus and fine-tuning budget so that frontier models and reference models are compared on identical items.
Frontier models will be evaluated zero-shot and few-shot, with matched prompt-tuning budgets. Two ablations isolate the key ingredients. The first removes cue masking, which tests whether contamination is easiest to see when date strings are left in the text. The second removes handwriting-recognition matching, which tests whether transcription quality alone produces a gap.
Why not just…?
Why not just use perplexity to detect contamination, as LatestEval does? Perplexity on historical text mostly reflects how badly a tokenizer fragments archaic spelling, which Rastas et al. and Kang et al. document. It is also unavailable for closed models. This is a real competitor, though, and the calibration models decide it. If the perplexity gap ranks known-contaminated against known-clean models as well as $C$ does, we will recommend perplexity and drop the behavioral benchmark.
Why not just reuse HUE's chronological-attribution task? HUE has no unpublished counterpart, so its scores cannot be decomposed into skill and memorization. Running our pipeline over HUE's own collections answers directly whether those reported scores are inflated.
Why not just have historians write new questions? Hand-written questions cost a great deal and become contaminated once published, as LatestEval and LiveAoPSBench show. Our generator can be rerun on newly transcribed pages whenever a benchmark goes stale. Historians remain in the loop only to audit faithfulness.
Why not just split by transcription date, treating pages put online after a model's cutoff as fresh? This is a serious alternative and largely a variant of our design. It fails when transcriptions are posted gradually or recycled, so we run it as an additional split. If it agrees with the publication-status split, it becomes the cheaper protocol.
Decisive Checks
On the calibration models, $C$ must be near zero for the clean reference and clearly positive for the deliberately contaminated copy. If it is not, the measure does not work and the project stops. On real models, the result is informative in either direction. A large $C$ for frontier models on the published Joseon records, with a small $C$ on the VOC handwriting-recognition material, would show that reported historical-language competence is partly recall. A $C$ near zero across all models would be the first evidence that this competence holds up on text no model has read.
References
- LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation
- When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation
- UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seeking
- Effective Distant Supervision for Temporal Relation Extraction
- Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in LLMs
- HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea
- Language Models Lack Temporal Generalization and Bigger is Not Better
- From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French
- OCR post-correction of the Royal Society Corpus based on the noisy channel model
- Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model
- Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation