Libraries digitize archives one collection at a time. A post-correction model, meaning a network that repairs the character errors an optical character recognition (OCR) engine leaves in scanned text, has to keep adapting as those collections arrive. A 1820s blackletter newspaper produces different errors than a 1950s Antiqua one. When a model is fine-tuned on a noisy new collection, two things go wrong. The first is familiar: it forgets the spellings and error patterns of earlier collections. The second is less discussed and more damaging: it learns to edit more often. It then rewrites text that was already correct, including archaic spellings, names, and clean modern lines. Prior work has shown that a one-step correction model can make low-noise text worse, raising the character error rate from 1.2% to 1.6% by altering 6% of correct characters (A Two-Step Approach for Automatic OCR Post-Correction). Nobody has studied what happens to that failure when the model learns from a stream.
We propose splitting the post-correction model by function and letting each part learn at a different speed. This borrows from complementary learning systems, a line of continual-learning work that keeps a fast-changing copy and a slow-changing copy of a network to trade off adaptation against retention. The decision whether to edit a character lives in a slowly consolidated gate. That decision depends on language-wide regularities and on the base rate that most characters are already right. The decision what to write instead lives in a fast-adapting editor, which absorbs era-specific confusions such as long-s read as f. The core claim is that overcorrection under distribution shift is mostly a gate problem, and that running the gate on a slow clock removes most of it without giving up the editor's plasticity.
Core Claim
Existing complementary-learning methods run fast and slow copies of the whole network. CLS-ER keeps a working model, a fast moving average of its weights, and a slow moving average, and replays buffered examples against whichever copy is more confident (Learning Fast, Learning Slow). The Linear Connector interpolates a stable and a plastic weight vector (Towards Better Plasticity-Stability Trade-off). Both were validated only on image classification. Separately, OCR work already splits detection from correction, but trains both parts offline on fixed corpora. The two-step German system does this with a BiLSTM detector (A Two-Step Approach), and the Bulgarian system does it with an mDeBERTaV3 detector at F1 0.90 (Post-ocr text correction for Bulgarian historical documents).
Our proposal assigns timescales to functions. We write the corrector's output at position $i$ as a mixture of copying and editing:
$$p(y_i \mid x) = g_{\text{slow}}(x, i)\, q_{\text{fast}}(y_i \mid x) + \big(1 - g_{\text{slow}}(x, i)\big)\,\mathbb{1}[y_i = x_i],$$
In this equation, $x$ is the OCR input and $y_i$ the output character. The gate $g_{\text{slow}}$ is the probability that position $i$ needs an edit. The distribution $q_{\text{fast}}$ gives candidate replacements, and the last term puts all probability on copying $x_i$ unchanged.
Both parts share a working model trained by gradient descent on the incoming stream. At inference, the gate is read from a slow exponential moving average of the working gate's weights, and the editor from a fast moving average of the working editor's weights. A small reservoir-sampled buffer of past corrected lines is replayed with a consistency loss, as in CLS-ER.
What is new is not a dual-speed network. It is the hypothesis that the right place to put stability in sequence correction is the edit decision, together with a test of when that hypothesis breaks.
Research Program
We build the corrector on ByT5, a byte-level pretrained transformer. Its fine-tuning on Swedish newspapers reduced character error rate from 3.20% to 2.04% while avoiding the subword-fragmentation problems noisy text causes (Post-OCR Correction of Digitized Swedish Newspapers with ByT5). A lightweight per-character gate head sits on the encoder. The editor is the decoder, conditioned so that it emits the copy distribution when the gate is closed.
The study then answers three questions in order. First, does naive sequential fine-tuning actually inflate the edit rate on earlier and cleaner data? We have no reported measurement of this. It is a prediction, not a known fact. We will measure false alteration of correct characters after each stream step.
Second, does a slow gate fix it, and at what cost? The obvious cost is lag. When a genuinely noisier era arrives, a slow gate under-edits until it catches up. We will measure that lag directly as recall of true errors over the first $k$ corrected lines of each new era.
Third, what happens when the correct answer itself changes? Bulgarian orthography moved from the Ivanchev to the Drinov convention, so text correct under one is "wrong" under the other (Post-ocr text correction for Bulgarian historical documents). Here a slow gate should hurt, because it keeps refusing edits that are now required. This is the setting where our claim is most likely to fail, and we treat it as the stress test rather than hiding it.
We also test one minimal fix for the lag and the orthography case: an asymmetric gate schedule that moves quickly when replayed error rates rise and slowly when they fall. This takes inspiration from the update modulation in C-MAML, which reacts to loss spikes at distribution shifts (Online Fast Adaptation and Knowledge Accumulation). We adopt it only if the plain slow gate fails the Bulgarian test.
Baselines
The naive reference point is copying the OCR output unchanged, which by definition alters no correct characters. Any method that raises character error rate over this on a stream step has failed on that step. The two-step paper shows neural correctors can do exactly that.
The sequential fine-tuning baseline is ByT5 fine-tuned era by era with no protection, using the same step budget. This isolates forgetting and edit-rate inflation.
The strongest existing post-correction method for the overcorrection problem is the two-step detect-then-correct design (A Two-Step Approach). On German historical text with 1.2% initial error rate, it reached 0.9% character error rate and altered only 0.3% of correct characters, versus 1.6% and 6% for a one-step model. We run it with a ByT5 editor and a transformer detector so that the comparison changes only the training schedule, not the architecture. It is trained two ways: sequentially with both parts plastic, and with the detector frozen after the first era. Beating it means lower final error averaged over all eras and lower false-alteration rate on early eras, with equal or better error recall on new eras.
The strongest continual-learning method is CLS-ER applied to the whole ByT5 corrector (Learning Fast, Learning Slow). On image classification it reports 23.47% accuracy on S-Tiny-ImageNet with a 200-sample buffer, versus 10.96% for DER++. We found no reported result on Lacuna for CLS-ER or any continual method on OCR post-correction. It gets the same buffer size, the same moving-average tuning budget (an identical grid over fast and slow rates), and the same total gradient steps as our method. This baseline is also our key ablation. It differs from our method only in whether the slow timescale is applied to the gate alone or to everything. If it matches us, the functional split adds nothing.
A plain replay baseline uses sequential fine-tuning with the same buffer and no moving averages.
The oracle is joint retraining from scratch on all data seen so far, as in the Swedish ByT5 data-mixture setup. It gets unlimited compute and represents the upper bound.
Why not just…?
Why not just retrain on everything whenever a new collection arrives? For corpora the size of the Swedish newspaper set (about 44,000 segments), this is affordable, and it is a genuine threat to the forgetting half of our story. We include it as the oracle and report our compute as a fraction of its compute. We do not expect joint training to solve overcorrection, though. A single model trained on a mixture of eras learns one average edit tendency, which is too aggressive for clean eras and too timid for noisy ones. If joint retraining matches our false-alteration rates at acceptable cost, we will say the contribution is limited to the online setting in which libraries correct a few lines at a time.
Why not just freeze the detector in a two-step system? This is the simplest version of "stable gate, plastic editor," and it may do most of the work. The Bulgarian orthography shift is where a frozen detector should break, because it never learns that formerly correct spellings now need editing. Comparing a frozen detector, a slow moving-average detector, and a fully plastic detector on the Ivanchev-to-Drinov stream settles whether slow consolidation beats freezing. If freezing wins everywhere, including on Bulgarian, we will report that and drop the moving-average machinery.
Why not just prompt a large language model and skip training? Zero-shot LLM correction is not reliably safe across languages. Llama-3.1-70B cut character error rate by 38.7% on 18th-century English but increased errors by 47% on historical Finnish, and only GPT-4o improved Finnish, by 11.9% (OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches). We include a prompted open-weight model with the same alignment-based overgeneration removal as a reference on each stream. We expect it to show the same era- and language-dependent overcorrection our gate targets.
Why not just condition the corrector on the OCR engine's confidence? Per-character confidence is a strong signal that most correctors ignore, as the Swedish ByT5 authors note. It is complementary rather than competing. The confidence-to-error relationship itself shifts across engines and eras, so it too needs a stable estimate. Where engine confidences are available, we add them as a gate input in all conditions, so that they do not confound the comparison.
Proposed Experiments
We build three era-ordered streams from the corpora in the file. The Swedish stream uses the National Library of Sweden newspapers, 1818–2018, split by decade block. The German stream uses Deutsches Textarchiv material ordered from the 17th to the 19th century, including the low-noise test condition of the two-step paper. The Bulgarian stream runs from ICDAR 2019 (Ivanchev orthography) to DOPOC (Drinov orthography). Within each era, lines arrive in small corrected batches to simulate human-in-the-loop digitization. Every era keeps a held-out test set.
After each stream step, we report character and word error rate on every era's test set. We also report the false-alteration rate on characters that were already correct, as defined by the two-step paper, and recall of true errors in the first few hundred lines of each new era to measure lag. Finally, we report backward transfer: the change in error on earlier eras by the end of the stream.
The decisive checks are three. The first tests the premise: if sequential fine-tuning does not inflate false alteration on early eras, the premise fails and we stop. The second is the ablation: the gate-only slow track must beat whole-model CLS-ER on false alteration at matched error recall, or the functional split is unjustified. The third is the Bulgarian stream: the method must not fall below the plastic two-step baseline after the orthography change. If it does, the asymmetric schedule is tested, and if that also fails, we will report that slow gating is unsafe when the definition of "correct" moves.
Beyond the headline scores, we will plot gate opening rate against true input error rate for each era over the stream. This directly shows whether the slow gate tracks base rates while the fast editor absorbs new confusion patterns.
References
- A Two-Step Approach for Automatic OCR Post-Correction
- Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
- Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector
- Post-ocr text correction for Bulgarian historical documents
- Post-OCR Correction of Digitized Swedish Newspapers with ByT5
- Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning.
- OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches