Transitivity-Gated Self-Training for Post-OCR Correction

When a scanned historical newspaper is converted to text, the character recognizer makes systematic mistakes, and a language model can be trained to fix them. Such training needs pairs of noisy machine output and careful human transcription, and those pairs are scarce. The obvious workaround is self-training: let the corrector fix a large pile of unlabeled scanned text, keep the fixes it is confident about, retrain on them, and repeat. That loop has a failure mode that the post-OCR literature and the model-collapse literature have each described, but separately. Post-OCR correctors already "over-correct" rare archaic spellings and valid proper names into common modern words (Post-OCR Correction of Digitized Swedish Newspapers with ByT5). Recursive training on a model's own confident outputs is known to erase exactly such low-probability tails, generation after generation (Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences). Self-training a corrector with a confidence filter should therefore drive the corrector to steadily modernize the archive, and its own confidence will not warn us.

We propose to filter the corrector's pseudo-labels with a different signal: whether its preferences among candidate readings are logically consistent. For each noisy line we gather several candidate corrections, always including the untouched OCR reading. We ask the model to compare every pair, and we score how far those pairwise judgments are from forming a coherent ranking, using the consistency ratio from the Analytic Hierarchy Process. The Analytic Hierarchy Process is a decision method that turns pairwise comparisons into weights and rejects judgment sets that contradict themselves. The bet is that a model guessing about an unfamiliar word will contradict itself across pairs (preferring A over B, B over C, and C over A) even when its single-candidate confidence is high. Lines whose comparisons are incoherent are kept as the original OCR reading or left out of training, and coherent lines become training targets.

Core Claim

Our central claim is that intransitivity in a model's pairwise preferences over candidate corrections detects "confident but wrong" corrections that scalar confidence misses. We further claim that gating recursive self-training on this signal preserves rare and archaic tokens across generations where confidence-gated self-training erodes them. The claim is falsifiable before any loop is run. On held-out lines with gold transcriptions, consistency ratio must predict whether the chosen correction is wrong better than log-probability confidence and better than plain ensemble disagreement, especially among high-confidence lines. If it does not, the project stops there.

Where This Sits in Prior Work

Three lines of work motivate the proposal. The closest anti-collapse method, Self-Verification Provably Prevents Model Collapse in Recursive Synthetic Training, keeps synthetic samples that fall within a margin of the best log-probability under a fixed anchor model trained on human data. It proves that collapse is bounded under a "confidence-calibrated agreement" assumption. That paper tests the method only on synthetic linear functions and math, and it names miscalibrated overconfidence as the failure case. Post-OCR correction of archaic text is a domain where that assumption plausibly fails, because a modern-trained model is confidently wrong about period spellings. We reuse its loop structure and replace the scalar gate with a structural one.

Model Collapse Despite Ground-Truth Verification shows theoretically that even a perfect verifier lets a model shed support over hard parts of the target distribution. This constrains us: a gate that only checks whether accepted outputs are right cannot prevent tail loss. Our design therefore treats "keep the OCR reading" as a first-class candidate, and it measures tail retention directly rather than only accuracy.

On the Dangers of Bootstrapping Generation for Continual Learning and Beyond shows that synthetic data can look indistinguishable from real data while being almost perfectly linearly separable in feature space. That motivates a representation-level collapse diagnostic in addition to error rates.

On the correction side, Leveraging LLMs for Post-OCR Correction of Historical Newspapers fine-tunes Llama 2 with low-rank adapters (LoRA) on the BLN600 corpus and reports the strongest single-pass numbers in the file. It flags hallucinated corrections as an open problem and suggests confidence estimation as the remedy, which is precisely the remedy we expect to fail. Multi-modular domain-tailored OCR post-correction combines statistical machine translation, spell-checkers and a vocabulary built from the text itself, using learned linear weights. It never checks whether its modules' preferences cohere.

The consistency machinery comes from Studies on Synthetic Outcomes Evaluation of Scientific Research, which gives the eigenvalue-based consistency ratio and the geometric-mean ("radical root") weight solver. On Multi-Class Cost-Sensitive Learning supplies a related insight. A matrix of pairwise ratios admits a single global weighting only when it is consistent, and when it is not, a scalar summary silently distorts it.

The nearest cross-direction system, A Multi-Model Self-Evolving Framework for Zero-Data Document Understanding via Axiomatic Synthetic Refinement, avoids collapse by rendering clean documents from code and filtering by a solver-accuracy band. It never touches degraded scans, and it never checks the coherence of the model's judgments. We work on real OCR noise with no rendering step.

Research Program

The consistency signal only exists if comparisons are made directly, and this is easy to get wrong. If each candidate receives a scalar score $s_i$ and we set $a_{ij}=s_i/s_j$, the resulting matrix is perfectly consistent by construction. The elementwise geometric mean of several such matrices is also perfectly consistent. Intransitivity can therefore only come from judgments in which the model sees both candidates at once.

For each OCR line we draw $n$ candidates, with $n$ around 5. These are the raw OCR reading plus $n-1$ sampled corrections from the current corrector. For every pair, we present the line, its surrounding context, and the two readings, and read off the model's probability of preferring each. Both presentation orders are averaged so that position bias does not masquerade as inconsistency. The odds are mapped to the usual 1/9 to 9 ratio scale to form a reciprocal matrix $A$, with $a_{ji}=1/a_{ij}$. We then compute

$$CR = \frac{(\lambda_{\max}-n)/(n-1)}{RI_n},$$

where $\lambda_{\max}$ is the largest eigenvalue of $A$ and $RI_n$ is the average value of the numerator for random reciprocal matrices of size $n$. We estimate $RI_n$ by simulation for our exact scale. The geometric-mean weights pick the preferred candidate.

The gate has three outcomes:

  1. Accept. If $CR$ is below a threshold and the preferred candidate differs from the OCR reading, the correction becomes a training target.
  2. Train on the OCR reading. If $CR$ is low and the OCR reading wins, the line enters training as-is. This explicitly teaches the model to leave correct text alone.
  3. Exclude. If $CR$ is high, the line is excluded from training, and at inference the original OCR reading is output. We also record which pairs contribute most to the inconsistency, to check whether contradictions concentrate on archaic forms and names.

The threshold is swept rather than fixed at the conventional 0.10, since that cutoff was calibrated for human judges.

The loop starts from a corrector trained on a small labeled seed. Each generation corrects an unlabeled pool of real OCR text, gates the results, retrains, and repeats for five generations. Variants with and without a fixed real-data anchor are run in parallel.

The cost is $n(n-1)/2$ pairwise calls per line per order, so about 20 calls at $n=5$. That is affordable for a pool of tens of thousands of lines, and we report compute alongside every comparison.

Baselines

The do-nothing reference is the raw OCR text. On BLN600 the Gale OCR text has a mean character error rate of 7% (BLN600: A Parallel Corpus of Machine/Human Transcribed Nineteenth Century Newspaper Texts). On the Swedish newspapers the uncorrected error rate is 3.20% (Post-OCR Correction of Digitized Swedish Newspapers with ByT5). Any method that raises the error rate on already-correct characters is penalized, regardless of its net error rate.

The second reference is the seed corrector without self-training. This shows whether any loop helps at all.

The strongest existing supervised method is a LoRA-tuned Llama 2 13B, which Leveraging LLMs for Post-OCR Correction of Historical Newspapers reports reduces character error rate on BLN600 by 54.51%, against 23.30% for BART large. That figure uses far more labeled BLN600 data than our seed. We therefore reproduce it on our split as an upper reference: full supervision on BLN600 minus the test excerpts. We also train the same model on only the seed, under an identical adapter configuration and tuning budget. Beating the seed-only model and closing a substantial fraction of the gap to full supervision, without raising the over-correction rate, counts as success. Matching full supervision would be a bonus.

The strongest competitor for the loop itself is anchor log-probability self-verification (Self-Verification Provably Prevents Model Collapse in Recursive Synthetic Training). We implement it with the seed corrector as the fixed anchor and the same candidate pool, and tune its margin over the same grid size as our threshold. Every gate is run at matched acceptance rates, so that no method wins merely by training on fewer or more lines.

Three ablations isolate the key ingredient:

  1. Scalar-matrix ablation. We build $A$ from the same model's per-candidate scores. This yields a consistent matrix by construction and reduces the gate to a weight margin, so it removes only the intransitivity information.
  2. Ensemble-disagreement gate. This gate uses how often the sampled candidates disagree.
  3. Random gate. Lines are accepted at random at matched rates.

Why not just…?

Why not just gate on the anchor model's confidence, as self-verification does? This is the main threat, and the proposal is built to test it head-on rather than assume it fails. The cited theory guarantees stability only when the anchor's confidence tracks correctness, and the over-correction of archaic forms documented for ByT5 suggests that is exactly what breaks here. Our first experiment measures how well each signal flags wrong corrections among the most confident decile of lines. If log-probability confidence matches consistency ratio there, the structural gate adds nothing and we would say so.

Why not just use agreement among several sampled corrections? Disagreement is cheaper and captures uncertainty. However, a model can produce the same confident modernization in every sample, giving perfect agreement and a wrong answer. Pairwise comparison against the untouched OCR reading forces the model to confront the alternative. The ensemble-disagreement gate is run at matched acceptance rate, so the experiments settle whether that confrontation matters.

Why not just train on clean period text with cheap injected noise and skip recursion entirely? Cheap Character Noise for OCR-Robust Multilingual Embeddings shows that random character corruption of 5% of text is a strong, cheap regularizer for embeddings. A corrector trained on clean text corrupted this way is a genuine competitor. Its limitation is that independent random flips do not reproduce the correlated glyph confusions of real scans, which the BLN600 authors attribute to physical degradation. We include it as a baseline and as an initialization, so that self-training on real noise must improve on top of it to count.

Why not just keep half the training data real, as the curation paper recommends? Mixing real data prevents collapse in the image experiments of Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences. Here, though, the real labeled pool is a few hundred excerpts, so a fixed 50% share means repeating the seed many times over. We run the anchored variant for every gate. If anchoring alone preserves tails as well as the consistency gate, the gate's value reduces to the zero-anchor regime, and we would report that narrower result.

Proposed Experiments

Data. BLN600 is split into a seed of roughly 100 excerpts, a development set, and a held-out test set of about 300 excerpts. The unlabeled pool is drawn from the same newspaper source, with gold used only for evaluation. A second setting uses the Swedish newspaper segments, with a ByT5 corrector replacing Llama 2, to test whether the effect holds for a character-level model and another language. ICDAR 2017 and 2019 serve only as secondary checks, given the contamination concern raised in Leveraging LLMs for Post-OCR Correction of Historical Newspapers.

Metrics. We report character and word error rates. We also report an over-correction rate: the fraction of characters that were correct in the OCR and made wrong by the corrector. A tail-retention score measures recall of gold tokens that are archaic spellings, period abbreviations, or proper names absent from a modern lexicon. Lexical diversity of outputs is tracked per generation. Following the bootstrapping paper, we also train a linear probe to separate the corrector's outputs from gold text in embedding space at each generation, as a collapse diagnostic.

Decisive check 1: before any loop is run. On development lines with gold transcriptions, we compute the area under the ROC curve for detecting a wrong selected correction. We compare consistency ratio, anchor log-probability margin, ensemble disagreement, and the scalar-matrix ablation, overall and within the top confidence decile. The project proceeds only if consistency ratio adds detection power beyond confidence there.

Decisive check 2: across five generations. We expect confidence-gated self-training to show rising tail loss and over-correction even while overall error rate looks flat or improving. We expect the consistency-gated loop to hold tail retention near its seed level. A result where all gates degrade at the same rate, or where the matched random gate performs as well, would refute the core claim.