Programs that clean up scanned text after optical character recognition (OCR, the software that turns page images into characters) make a specific, costly mistake. Shown a rare but correct word, such as an archaic spelling, a local place name or a trade term, they "fix" it into something more common. On nearly clean historical text, a standard neural corrector changed 6% of correct characters and raised the character error rate from 1.2% to 1.6% (A Two-Step Approach for Automatic OCR Post-Correction). These correctors look only at a line or a sentence. The question they get wrong, "is this odd-looking token an error, or a legitimate word this document would use?", is about the whole document, not the local window.
We propose to answer that question with a small, explicit document-level prior: a probability over words, estimated from the noisy document itself, that the corrector consults before changing anything and while choosing a replacement. The prior has two parts. The first is a cache of word forms seen in the document. The second is a topic model, which spreads probability to words the document's themes predict even when they appear only once or not at all. Topic models are a reasonable source for this signal because document-level themes survive raw OCR noise: LDA run on uncorrected Texas newspapers still recovered coherent historical topics (Topic Modeling on Historical Newspapers). The claim is falsifiable. Topic smoothing should help most where the cache helps least, namely for gold words that occur once or never elsewhere in the document. A prior taken from a different, randomly chosen document should erase the gain.
Core Claim
Local post-correctors are nearsighted. The fix is not more raw context, which only adds more corrupted tokens to the model's input. The fix is a compressed, noise-tolerant summary of what the document is about, used in two places. The first is the error detector's decision to touch a token. The second is the scoring of candidate corrections.
The new element is not "use document context." Prior systems already do that in narrower forms, as described below. What is new is the claim that a topic-smoothed prior covers the low-frequency words that a flat document cache misses, together with a test design that can show whether this is true.
Relation to Prior Work
The motivating evidence comes from two places. The two-step paper above shows that putting an error detector in front of a sequence-to-sequence corrector cuts false edits to 0.3% of correct characters. Its detector, however, sees only local patterns, so rare valid words remain exposed. Post-ocr text correction for Bulgarian historical documents uses the same detect-then-correct design with an mDeBERTaV3 detector (F1 0.90). It is equally local.
The closest prior work uses document-wide or corpus-wide lexicons without any topic structure. Multi-modular domain-tailored OCR post-correction adds a text-internal list of high-frequency words from the book being corrected. That list is a flat frequency cutoff, so words appearing once or twice never enter it. Lexically Aware Semi-Supervised Learning for OCR Post-Correction constrains beam search with a weighted finite-state automaton built from corpus-wide frequencies. It applies the same distribution to a legal document and an agricultural one. Our prior is document-specific and smoothed by topics, and we compare against both styles directly.
The method itself comes from topic-mixture language modeling (PLSA Based Topic Mixture Language Modeling Approach). That work conditions word prediction on a document topic vector. We reuse the mixture formula as a decoding prior rather than retraining the corrector around it.
We deliberately avoid the heavier route of Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables. That paper injects topic codebooks into attention layers and reports roughly 3× the latency of a plain Transformer. Our prior enters only at scoring time, so it adds little cost and can sit on top of any existing corrector.
Two papers set limits on the design. OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches finds that longer context windows help large language models (LLMs) on English but cannot stop failures on Finnish. DAGS: A Dependency-Based Dual-Attention and Global Semantic Improvement Framework for Metaphor Recognition finds, in a different task, that adding surrounding words can hurt and pruning them can help. Both suggest that global information should arrive as a summary, not as more raw tokens. We test this rather than assume it.
Research Program
The base system is a two-step pipeline. A byte-level detector flags suspect tokens. A ByT5 corrector, as in Post-OCR Correction of Digitized Swedish Newspapers with ByT5, rewrites only the flagged spans.
For each document $d$ we build a word prior:
$$p_d(w) = \alpha\, c_d(w) + (1-\alpha) \sum_{k=1}^{K} \theta_{d,k}\, \phi_k(w).$$
The terms are defined as follows. $c_d(w)$ is the smoothed relative frequency of word $w$ in the document's own noisy text. $\theta_{d,k}$ is the document's weight on topic $k$. $\phi_k(w)$ is topic $k$'s word distribution, learned on the noisy training archive. $\alpha$ is tuned on development data.
OCR splits one word into many surface variants. Both the cache and the topic model therefore operate on clusters of variants, grouped by character-n-gram similarity, so that variants share evidence.
The prior enters in two places. First, the detector receives $\log p_d$ of the current token and of its best nearby candidate as extra features. Second, the corrector's beam search scores each hypothesis $y$ as
$$\log p_{\text{corr}}(y \mid x) + \lambda \log p_d(y),$$
where $x$ is the noisy input span and $\lambda$ is tuned on development data.
We also test one round of closed-loop refinement: correct the document, re-estimate $\theta_d$ and $c_d$ from the corrected text, and correct again. We measure whether this compounds errors.
For articles inside multi-article newspaper pages, we compare two definitions of "document": the whole page, and article units recovered with the layout-graph method of Post-OCR Paragraph Recognition by Graph Convolutional Networks. This tells us how well the global summary must match the real discourse unit.
Baselines
The naive reference point is the raw OCR output with no correction. A cautious method has to stay close to this when the input is already clean. On the OCR-D German Deutsches Textarchiv data the raw error rate is 4.3% CER (A Two-Step Approach for Automatic OCR Post-Correction). On the Swedish newspaper set it is 3.20% CER.
The first trained baseline is a one-step ByT5 corrector applied to every line. It is the standard design, and it is known to over-correct on clean text.
The strongest existing method on our core question is the detector-gated two-step pipeline. It reports a final CER of 0.9% on its German test set and false modification of only 0.3% of correct characters. The paper also describes this as an 18.2% relative improvement, which does not match the stated CERs; we will recompute both figures on our own split. On Swedish newspapers, the strongest reported single model is ByT5 fine-tuned on newspapers plus Blackletter, at 2.04% CER (Post-OCR Correction of Digitized Swedish Newspapers with ByT5). We re-implement both systems on identical splits, using the same backbone, training data and hyperparameter search budget as our system. The only difference is the document prior.
The LLM reference is Llama-3.1-70B and GPT-4o with up to 300 words of preceding context, following OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches. That paper reports a 38.7% CER reduction on English ECCO-TCP, a 47% CER increase on Finnish for Llama-3.1-70B, and an 11.9% reduction on Finnish for GPT-4o.
The ablations isolate the key ingredient. We run four variants: cache only ($\alpha=1$), topics only ($\alpha=0$), an oracle prior estimated from clean gold text, and a mismatched prior taken from a random other document. Beating the two-step pipeline means lower CER together with a false-modification rate no worse than its 0.3%, with the gain concentrated on rare-in-document gold words. Matching it overall while lowering false modifications would also count.
Why not just…?
Why not just use a document cache, as Schulz and Kuhn did? This is the real threat. A cache of the document's own word forms may capture most of the benefit. We include it as a full arm of the comparison, not a straw man. The experiment that settles the question stratifies errors by how often the gold word appears elsewhere in the document. If topic smoothing adds nothing for words appearing once or never, the topic component is not worth having, and the contribution reduces to a better-engineered neural cache. We would report that outcome as the result.
Why not just feed the corrector more raw context? We give ByT5 the preceding and following lines up to the same token budget that the prior summarizes. The Kanerva et al. results show that extra context helps English LLMs but does not prevent failures on non-English text. If matched raw context equals the prior on low-CER German and Swedish text, the summary hypothesis is wrong.
Why not just prompt a large LLM with the whole page? The Finnish result, a 47% CER increase for Llama-3.1-70B, together with the reported tendency of LLMs to modernize archaic spellings, are exactly the over-correction we are targeting. We still run the LLM baselines on ECCO-TCP and NLF Finnish. On English, an LLM may simply win on CER. In that case we report false-modification rates side by side, because that metric is the one this proposal claims to improve.
Why not just train a stronger local detector? We give the two-step baseline the same extra training compute. If its false flags on rare valid words fall as much as ours do, the gain came from capacity, not from global context.
Proposed Experiments
The main testbeds are OCR-D German DTA, chosen for its low-CER regime and published false-modification figures, and the Swedish newspaper set, chosen for multi-article pages. We add ECCO-TCP and NLF Finnish to check the LLM comparison and to test across languages.
We report CER and WER. We also report the false-modification rate on correct characters, the rate of real-word substitutions (corrections that produce a valid but wrong word), and all errors broken down by the gold word's in-document frequency and by whether it is out of vocabulary.
Three checks decide the outcome. The first is the mismatched-prior control: if a random document's prior helps as much as the true one, the gain is generic smoothing, not document context. The second is the frequency stratification, which separates topic smoothing from caching. The third is the gap between the oracle prior and the noisy-estimated prior as we inject synthetic noise at increasing rates, using the realistic text-reuse noise of Leveraging Text Repetitions and Denoising Autoencoders in OCR Post-correction. This curve shows how much page damage the global estimate can tolerate before it stops helping.
As a secondary probe, we test whether topic compatibility breaks the "long s" false consensus on the Text Creation Partnership data. This is the case where every duplicate witness makes the same systematic error (Multi-Input Attention for Unsupervised OCR Correction).
References
- A Two-Step Approach for Automatic OCR Post-Correction
- Topic Modeling on Historical Newspapers
- Post-ocr text correction for Bulgarian historical documents
- Multi-modular domain-tailored OCR post-correction
- Lexically Aware Semi-Supervised Learning for OCR Post-Correction
- PLSA Based Topic Mixture Language Modeling Approach
- Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables
- OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches
- DAGS: A Dependency-Based Dual-Attention and Global Semantic Improvement Framework for Metaphor Recognition
- Post-OCR Correction of Digitized Swedish Newspapers with ByT5
- Post-OCR Paragraph Recognition by Graph Convolutional Networks
- Leveraging Text Repetitions and Denoising Autoencoders in OCR Post-correction
- Multi-Input Attention for Unsupervised OCR Correction