Correction Programs, Not Corrected Text: Auditing LLM Repair of Historical OCR for Memorization

Large language models are poor at directly fixing the errors that optical character recognition (OCR) leaves in scanned historical documents. Asked to clean up a noisy 19th-century newspaper line, they paraphrase, modernize spelling, and invent sentences. We propose a different arrangement. The model writes a small correction program, a Python function that maps each OCR line to a corrected line using bounded edit operations. The program runs on a few hundred labeled lines, and the model debugs it over several rounds using the concrete diffs where the program failed or made things worse. The model therefore never emits the historical text itself. Its only influence on the output is code that can be executed, inspected, and constrained.

The same arrangement answers a measurement question that the benchmark contamination literature has left open. Contamination here means that test data leaked into a model's training corpus. When iterative refinement with execution feedback "works", is the model reasoning from the feedback, or is the feedback cueing recall of text it has seen? Historical corpora give an unusually clean natural experiment for this. Some post-correction benchmarks are drawn from public-domain literature that is almost certainly in pretraining data. Others come from regional newspaper archives that were never on the web. Because the model's contribution is a program, we can look inside it for memorized target strings and cap how much literal text it may contain. That turns contamination from a suspicion into a quantity we can measure.

Core Claim

Our claim is that program-mediated iterative correction has three properties. It yields positive post-correction improvement on historical OCR where direct LLM rewriting yields negative improvement. It reaches this in the low-label regime, around 100 to 500 aligned lines, where fine-tuning is weakest. And its gains transfer almost fully from likely-contaminated to uncontaminated corpora. Direct rewriting, by contrast, should show a large "contamination premium", meaning much better scores on memorized text than on unseen text. Whether that premium exists, and how large it is, is the empirical question. The literal-budget constraint described below is the tool that isolates it.

The work sits between three findings. The first motivates it: Post-Correction of Historical Text Transcripts with Large Language Models: An Exploratory Study evaluated 14 LLMs on eight historical datasets in single-pass prompting and found consistently negative Post-Correction Improvement Scores (PCIS), with hallucination and modernization as the main failure modes. The second also motivates it: Leveraging LLMs for Post-OCR Correction of Historical Newspapers abandoned the ICDAR 2017/2019 benchmarks because their literary excerpts were probably in Llama 2's pretraining data, and evaluated only on BLN600. It treated contamination as something to avoid, not something to measure. We turn that same contrast into an experimental variable.

The third finding constrains the design. Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models and Test of Time show that contamination inflates code benchmarks and that rephrasing hides it from surface detectors. Both studied single-turn generation only. Neither asked what a multi-turn feedback loop does with memorized material. The research file found no prior work that frames historical normalization or post-OCR correction as iterative program synthesis.

Research Program

We will build a correction agent. At each round, the model produces a function correct(line) -> line. It writes this against a small library that includes character substitution rules, context-conditioned regular expressions, whitespace merge and split operators (which target the word-segmentation errors that Beshirov et al. identify as hard for character-level models), and a lexicon built only from the training split.

A sandbox runs the function on the labeled training lines and returns three things: character error rate (CER), a sample of the worst diffs, and a list of lines the program made worse. If a revision raises regressions on the training split, it is rolled back. This mechanism is borrowed from the regression-reverting hardware agents of Fang et al.. The rollback exists because Yan et al. found that compiler-feedback loops plateau after one or two rounds and then oscillate, and because SlopCodeBench documents constraints being lost across iterations. We will also test Thompson-sampling search over candidate programs (From Search to Policy) as a stronger search procedure than a single linear chain of revisions.

The key instrument is the literal budget. A static check limits the total length of string literals in the program that do not appear in the training split. Under a tight budget, the program cannot carry memorized target text. It can only encode rules. Under an unlimited budget, a model that remembers the benchmark's ground truth can smuggle it in as a lookup table. The size of the gain from relaxing the budget, measured separately on each corpus, is our direct measure of recall. We will confirm it by scanning the programs for long literals that match held-out test targets.

We will split the corpora by how likely they are to be memorized. The likely-contaminated side is ICDAR 2017 and ICDAR 2019, which are literary and monograph excerpts. The likely-clean side is BLN600 (19th-century British newspapers), DOPOC (Bulgarian newspapers in the pre-reform Drinov orthography), and post-correction of the LSTM output on Chronicling Germany, including its 20 out-of-distribution pages.

We will not rely on the published assumptions about which side is which. For an open-corpus model (OLMo2, the model used for influence tracing in Test of Time), we will run direct n-gram and normalized-Levenshtein search of ground-truth lines against its training corpus, following Riddell et al. For all models, we will compute a perplexity-based membership score on the ground-truth text, in the spirit of Yax et al.. Lines will then be binned from high to low membership within each corpus. This matters because the within-corpus comparison removes differences in difficulty between corpora. As a secondary setting, we will run the same agent on word-level spelling normalization using Bollmann's ten-dataset corpus from A Large-Scale Comparison of Historical Text Normalization Systems. Programs there can use sentence context, which is how we will attack the homograph cases that token-level systems cannot resolve.

Baselines

The naive reference point is to leave the OCR output unchanged. This scores zero on PCIS and zero percent CER reduction. Any method must beat it on every corpus. On this point the field already fails, since direct LLM correction scores negative PCIS (Exploratory Study).

The second baseline is direct LLM correction, prompted zero-shot and few-shot at the sentence level, which that study found least harmful. We will use the same models that write our programs. This is the specific failure the proposal is designed to fix, and the file reports only that PCIS was consistently negative, not a single value.

The strongest existing method on clean newspapers is LoRA instruction tuning of Llama 2 13B. Thomas et al. report a 54.51% CER reduction on BLN600 with it, against 23.30% for fine-tuned BART-large. We will retrain this baseline on exactly the same labeled subsets used as the program's training split, at 100, 250, 500 and the full training set. We will tune its hyperparameters with a budget matched in GPU-hours to the agent's inference cost. Beating it means one of two things. Either the program matches or exceeds its CER reduction at 500 lines or fewer, or it has a clearly lower rate of degraded lines at comparable CER reduction on the full set. We do not expect the program to win on raw CER with full data, and we will report it if it does not.

For Bulgarian, the strongest specialist baseline is the detector-plus-seq2seq pipeline of Beshirov et al.. It reports a 25.4% improvement on ICDAR 2019 Bulgarian and 26.9% on DOPOC. We will run it with its released configuration on our splits. For normalization, the baseline is the character-level statistical machine translation system cSMTiser, which Bollmann reports at 90 to 95% word accuracy across most datasets. We will compare the two separately on the homograph subset, where our programs have a structural advantage.

Two ablations isolate what is new. The first is iterative self-refinement on text: the same model, the same feedback of diffs and regressions, and the same round and token budget, but it revises the corrected text instead of a program. This separates "program as output" from "iteration with feedback". The second is a single-shot program with no refinement rounds, which isolates the contribution of the loop. The literal-budget sweep acts as a third ablation, aimed specifically at memorization.

Why not just…?

Why not just fine-tune an LLM on the historical data, since fine-tuning washes out contamination? Rethinking the effects of data contamination in Code Intelligence found the "reset" effect only when fine-tuning on around 100,000 samples. With 500 samples, the benefit from pretraining leakage persisted. Historical collections are usually small, and Bollmann shows neural models learn steeply only once they have more tokens than typical datasets provide. Even so, this baseline is a real threat on BLN600, given its reported 54.51% CER reduction. The label-count sweep will settle it: if LoRA matches the program at every size, the program's remaining case rests on auditability and on not degrading lines, and we will say so.

Why not just let the LLM iteratively self-correct the text using the same feedback? This is the most serious alternative, because it keeps the loop and drops only the code. Our text self-refinement ablation tests it with matched compute. We expect it to inherit the hallucination and modernization problems, and to show a larger gap between ICDAR and the clean corpora, because nothing stops it from reproducing remembered text. If it matches the program on the clean corpora with a similar degraded-line rate, then the program is not what drives the gain, and the contribution shrinks to the contamination audit alone.

Why not just use cSMTiser or Norma, which already reach 90–95%? For isolated-token normalization with enough data, these systems are hard to beat, and we will not claim otherwise. Bollmann notes that they ignore context, so homographs fail, and they cannot repair word merges and splits. Our comparison restricted to homographs and segmentation errors will show whether context-aware programs close that gap.

Why not just measure contamination with n-gram overlap and skip the program machinery? For the open OLMo2 corpus, we do exactly that, as ground truth. But overlap measures only whether text was seen, not whether a correction method exploits it. Test of Time also shows surface signals miss rephrased leakage. The literal-budget sweep measures exploitation directly, and it works for closed models whose training corpora are unavailable.

Proposed Experiments and Decisive Checks

The primary experiment crosses five factors. The method is one of the baselines above. The corpus is ICDAR 2017, ICDAR 2019, BLN600, DOPOC, or Chronicling Germany. The label budget is 100, 250, 500 lines, or the full set. The model is OLMo2 and one stronger code model. The literal budget is tight or unlimited. We will report CER reduction, PCIS, and the fraction of test lines made worse, with bootstrap confidence intervals over lines.

Four results would be decisive. First, the program agent must post positive PCIS on every corpus with a degraded-line rate near zero. Failing that would falsify the central claim. Second, we will compute the contamination premium as the gain on high-membership lines minus the gain on low-membership lines within each corpus. This premium should be large for direct and text-refined correction and small for tightly budgeted programs. It should then grow when the literal budget is lifted, and the scanned programs should show that growth coming from embedded test strings. If lifting the budget does nothing on ICDAR, recall is not the mechanism and we will report that. Third, we will plot refinement curves over rounds, split by membership bin. These test whether the plateau Yan et al. observed arrives later on memorized lines, which would mean feedback partly acts as a cue for recall. Fourth, the out-of-distribution pages of Chronicling Germany test whether programs refined on one set of publications carry over to unseen ones better than the fine-tuned model does. Programs encode explicit, readable rules, so any failure to transfer can be inspected rule by rule.