Scannos as Accidental Laundering: Contamination in Public-Domain Pre-training Data

Openly licensed pre-training corpora are built largely from old, scanned books, newspapers, and government reports. These corpora exist to reduce copyright liability. Their text passes through optical character recognition (OCR, software that turns page images into text), which leaves character-level errors called "scannos", such as "tbe" for "the". This proposal asks whether that noise has a side effect nobody has measured. When a benchmark question or a copyrighted passage appears in such a corpus in corrupted form, the corruption may blind the tools that check for training-data leakage while leaving the model's benefit from seeing the text largely intact. The same corpora are often cleaned by a large language model that rewrites OCR output. That model was itself trained on unlicensed web data, and it may reinsert memorized wording, including benchmark text, into a corpus that was assembled to be clean.

The core claim is measurable. As the character error rate of leaked text rises, contamination detectors lose their signal faster than the model loses its benefit. We call the difference the detection–benefit gap. We also claim that LLM-based OCR correction changes this gap in a way that depends on what the corrector memorized, which can be tested by comparing items the corrector could have seen with items published after its training cutoff. Both effects would mean that "ethical" corpora carry a contamination and provenance risk that standard audits cannot see.

Where this sits

The closest work is Test of Time. It shows that semantic rewriting "launders" contamination. When arXiv theorems are turned into reasoning questions, or when LiveCodeBench variable names are perturbed, the post-cutoff performance drop disappears. Influence tracing on OLMo2-7B also fails to find the source document: the source ranked first 77.5% of the time for verbatim cloze questions but only 17.5% for transformed ones. That laundering is deliberate and meaning-preserving, and it is applied to clean digital text.

Our setting differs in two ways. The corruption is character-level, not semantic. It also arises passively, in exactly the archives that permissive corpora depend on, without any adversary. We reuse Test of Time's influence-function method as our mechanistic check.

Two other papers motivate the detection side. Soft Contamination Means Benchmarks Test Shallow Generalization finds semantic duplicates that n-gram decontamination misses, using embedding cosine similarity plus LLM verification. It also shows that very low contamination rates, around 4 per 10,000 fine-tuning examples, yield roughly 15% benchmark gains. Cheap Character Noise for OCR-Robust Multilingual Embeddings shows that scannos fragment subword tokens and break embedding retrieval, which suggests that embedding-based contamination detectors will break the same way.

The same paper also supplies a method. Fine-tuning on 5% randomly noised text restores retrieval, for example by more than 4 points for M-GTE on Blackletter noise. We use this to build a noise-aware detector.

Common Corpus supplies the regime. It is 2T tokens of public-domain and openly licensed text, cleaned with OCRonos, a fine-tuned Llama-3 8B rewriter. Its authors state that residual OCR noise is unavoidable and that rewriters can hallucinate or over-correct. None of its evaluations examine contamination.

Research Program

We run controlled pre-training in the style of Common Corpus, where we know exactly what the model saw. The base corpus is a 10–20B-token slice of Common Corpus's Open Culture and Open Government pillars. Into it we plant evaluation items at known repetition counts of 1, 4, and 16 copies. Each benchmark is split into a planted half and a matched unplanted half, following the seen-versus-unseen design of Soft Contamination.

Each planted item is inserted in one of three forms. The first is clean. The second is OCR-noised at character error rates of 2, 5, 10, and 20%. The noise is sampled from confusion statistics estimated from real historical newspapers (the HISTLUX Luxembourgish scans used in Cheap Character Noise), with a Blackletter-style variant. The third form is noised and then corrected by an LLM rewriter.

We train 350M and 1.2B models to match the PleIAs scales, with one tokenizer across all conditions. For each condition we then measure two quantities on the same items. Benefit is the planted-minus-unplanted accuracy or likelihood gain. Detection is the rate at which each detector flags planted items at a fixed false-positive rate on unplanted items. The gap curve across error rates is the main result.

The second strand audits the corrector and uses temporal isolation as a control. We draw planted items from sources dated before and after the corrector's training cutoff:

Post-cutoff items cannot be in the corrector's memory. The test is therefore simple. If corrected pre-cutoff items move closer to their canonical benchmark wording than the noise alone can explain, while post-cutoff items do not, the corrector is reinstating memorized text. The same test, run on passages from modern copyrighted editions of public-domain works, measures whether correction completes damaged spans with protected wording. That is a direct provenance risk for a corpus assembled to avoid liability.

To isolate the cause, we compare OCRonos with a corrector of the same size trained only on permissive text, using synthetic noise over Common Corpus.

Baselines

The naive reference point is the clean-planted condition. Its detection rate shows what each detector can do when the text is intact. Its benefit is the maximum contamination effect. Unplanted items give chance-level detection and zero benefit.

The first detector is exact 13-gram overlap, the standard decontamination filter. It is expected to fail quickly under noise.

The strongest existing detector is the embedding-similarity pipeline with LLM verification from Soft Contamination. It is the only method in the file shown to recover non-verbatim duplicates at corpus scale. We run it with its published embedding model and a verification prompt tuned on the clean condition only, so it is not tuned toward noise. The file reports prevalence figures for this pipeline (semantic duplicates for 78% of CodeForces problems in Olmo3 data) but no detection accuracy against ground truth. Our planted design supplies that measurement for the first time.

We also run Min-K% Prob membership inference, as used by Unveiling Causal Reasoning. No reported detection AUC was found on Lacuna for Min-K% on any of these benchmarks.

The mechanistic reference is EK-FAC influence tracing via Kronfluence, following Test of Time. Its reported hit rate of 77.5% on cloze items is the number to compare against on our clean condition.

Our key ingredient is a noise-aware detector: the Soft Contamination pipeline with its embedding model fine-tuned on 5% character-noised text, per Cheap Character Noise. Two ablations isolate what that fine-tuning adds. The first is plain character-level fuzzy matching by edit-distance alignment. The second is the same embedding pipeline without noise fine-tuning. All detectors receive the same corpus shards and are compared at a 1% false-positive rate calibrated on unplanted items. The detection-side claim is supported only if noise-aware detection recovers, at 10% character error, at least most of the clean-condition detection rate that the standard embedding pipeline loses.

Why not just…?

Why not just run OCR correction before decontamination? Correction does restore surface overlap, but the file says it cannot be complete (Common Corpus), and the corrector is itself a source of risk. Our noised-then-corrected condition tests this directly. If OCRonos-corrected pre-cutoff items drift toward canonical benchmark wording while post-cutoff items do not, then correcting first converts an invisible leak into an injected one, and it does so selectively for text the corrector memorized.

Why not just use fuzzy character matching? This is the real threat to the detection claim, because edit-distance alignment is cheap and noise-tolerant by construction. We run it as a first-class baseline at every error rate. If it closes the gap on its own, we will report that and drop the noise-tuned embedding detector as unnecessary. The benefit-side measurement and the corrector audit remain the contribution. The case where fuzzy matching should struggle is noise combined with paraphrase, and we include a small paraphrased-and-noised condition to probe it.

Why not just evaluate only on post-cutoff data? Temporal isolation avoids leakage only for benchmarks. It says nothing about whether a public-domain corpus contains noisy copies of older items. Test of Time also shows the post-cutoff signal is fragile. In our design, temporal dating serves a different purpose: it is the control that separates corrector memorization from noise.

Proposed Experiments and Decisive Checks

The first decisive result is the pair of curves, benefit versus character error rate and detection versus character error rate, for each detector at both model scales. The hypothesis fails if detection falls no faster than benefit. It also fails if benefit collapses at the same error rate at which detection fails, which would mean noisy leakage is harmless.

The second check is mechanistic. We run Kronfluence on the 350M model and ask whether the noisy planted document still ranks first among influential training points at rates near the clean condition, even as detectors miss it. That result would show the model is actually using text that audits cannot see.

The third is the corrector audit. We report the normalized edit distance from corrected text to canonical wording, split by pre- versus post-cutoff source. We compare OCRonos with the corrector trained only on permissive text. For the copyrighted-edition passages, we also report the verbatim reinstatement rate.

As a sanity check, clean-text performance on XCOPA and XStoryCloze, the benchmarks Common Corpus used, should be unchanged across planting conditions. This confirms that the planted items did not distort general training.