Sparse Visual Proofreading for Historical OCR

Machine transcription of old books and newspapers fails in a specific way. The errors are not spread evenly across the page. They cluster on a small number of glyphs that look alike but mean different things: the archaic "long s" (ſ) against f, worn Fraktur letters, ligatures, characters half-hidden by ink bleeding through from the back of the page. On these glyphs the system's knowledge of modern language pulls it toward the familiar reading. On an 18th-century French novel, a commercial OCR engine made 35.93 errors per page, and 26.29 of those were "modernizations" that turned archaic forms into modern ones (Digitizing 18th-Century French Literature). Human transcribers reading without care showed the same bias. A spelling-correction model that never sees the page cannot fix this, because both readings are plausible words. The model that does see the page also tends to trust its language prior over the pixels.

We propose a second pass that reads the page again, but only where it matters. A first-pass historical transcriber produces a draft. A lightweight proofreader then looks at the draft token by token. For most tokens it chooses a "copy" option that costs almost nothing. For the few ambiguous tokens it hands the work to specialist modules, which look back at a high-resolution crop of that glyph and decide between competing readings. Each specialist covers one family of visual confusions. This design is a mixture of experts, meaning a layer made of several parallel sub-networks plus a small gating network that picks which sub-network handles each token. The gating network here decides from first-pass uncertainty and local degradation, not from the task label. The claim is that this targeted look-back removes modernization errors that dense page-reading models still make, at a small fraction of the cost of re-reading everything.

Core Claim

Recent historical recognizers read pixels directly. CHURRO, a 3B vision-language model fine-tuned on 155 historical corpora, reaches 82.3% normalized Levenshtein similarity on printed and 70.1% on handwritten pages (CHURRO). Even so, we hypothesize that its errors on diplomatic transcription, meaning transcription that preserves original spellings and glyphs, are dominated by a small set of visually ambiguous spans. On those spans the decoder's language prior overrides the image. This is the "visual laziness" that Ive et al. found in multimodal translation and fixed with a draft-then-proofread deliberation decoder (Distilling Translations with Visual Awareness).

Our claim has two parts. First, a proofreader that is explicitly trained to contrast a draft against glyph crops, and that is sparsely invoked, fixes these spans better than either a text-only corrector or a dense model re-reading the page. Second, sparse invocation is not just a speedup. Because the proofreader leaves confident tokens untouched, it avoids "correcting" archaic forms that were already right, which is a known failure of post-correctors.

The closest prior system is NeKo (NeKo). It uses task-guided mixture-of-experts routing to correct OCR output and cuts word error rate on historical PleIAs text from 71.03% to 14.43%. NeKo differs from our proposal in two ways. It is text-only, so it cannot look at the glyph. It also routes by task (all OCR tokens go to an OCR expert), whereas we route by per-token visual ambiguity. CHURRO differs in the opposite direction: it sees the page but is dense and single-pass, with no mechanism that forces it to re-examine pixels when its prior is confident. Ive et al. supply the deliberation idea but use dense decoders on clean photographs.

Research Program

The first-pass model is CHURRO, used as released, or our own re-run of its Qwen 2.5 VL fine-tuning recipe on CHURRO-DS. We run it line by line on line crops, which gives a per-token probability distribution and an approximate image region for each token. The region comes from cross-attention maps, refined by character-width estimates within the line.

The proofreader is a small decoder. Its inputs are the draft line, the first-pass token entropies, and visual features from the line crop. It has one sparse layer per block, containing three kinds of module:

  • a null expert that copies the draft token;
  • an always-active shared expert that reads general line context, following the shared-expert pattern that Li found carries basic features in heterogeneous mixture-of-experts models (Decoding Knowledge Attribution in Mixture-of-Experts);
  • a handful of routed experts that cross-attend to a twice-upsampled crop around the token.

We build the proofreader by upcycling a dense decoder through residual factorization. Each expert is a frozen base plus an input-dependent offset, which Wu et al. showed costs under 3% extra training overhead (Residual Mixture of Experts).

Training proceeds in two phases:

  1. Guided routing. We use NeKo's guided-routing trick at a finer grain. We align first-pass drafts to ground truth on the training split and label each error with a confusion family: long-s versus f, Fraktur minim letters, ligatures, diacritic loss, and bleed-through phantom strokes. During early training, tokens with a family label are forced to that family's expert.
  2. Free routing. Routing is then released to the learned gating network, with a load-balancing term and a compute penalty on non-null routing.

Training data has two sources. The first is real first-pass errors on held-in CHURRO-DS corpora whose guidelines are diplomatic. The second is a glyph-swap augmentation: we splice same-font alternative glyphs harvested from the same page into real words. This produces pairs in which context is constant and only the pixels decide the answer.

Two design constraints come from prior negative results. Li reports that shallow mixture-of-experts models lose 76% of performance when their top experts are blocked. We therefore keep the shared expert and the null expert, so that no single routed expert is a point of failure. Wu et al. report a 6–8% latency penalty from sparse dispatch in vision backbones. We therefore measure wall-clock time, not only the fraction of tokens routed.

Baselines

Naive reference. The do-nothing reference is the first-pass CHURRO transcription itself. On CHURRO-DS its reported scores are 82.3% normalized Levenshtein similarity on printed text and 70.1% on handwritten text (CHURRO). We will reproduce these numbers on our split before reporting any gain. Beating this reference means a statistically reliable gain over our reproduced score, using a paired bootstrap over pages. The gain must appear especially on the glyph-preservation metric described below.

Strongest existing method. The strongest competitor is a dense second pass. This is a deliberation decoder in the style of Ive et al., with the same parameter count as our full proofreader. It receives the same draft and crops but processes every token. It is the right adversary because it isolates whether sparsity and specialization add anything beyond simply looking again. It is trained on identical data, with the same number of optimizer steps and the same hyperparameter search budget of 20 trials per method.

Other baselines and their fair-run conditions.

  • Dense fine-tuning. Continue fine-tuning CHURRO's own weights for the proofreader's training budget, on the same diplomatic subsets plus the glyph-swap data. This tests whether extra dense supervision alone suffices.
  • Text-only corrector. A NeKo-style mixture-of-experts corrector trained on the same drafts with the image removed.
  • Classical pipeline. On Chronicling Germany, the authors' own pipeline, whose fine-tuned LSTM line recognizer reaches a Levenshtein distance of 0.025 on pre-segmented lines (Chronicling Germany).

Ablations of the key ingredient.

  • Replace the uncertainty-based gating with random routing at the same invocation rate.
  • Replace it with task-level routing, where all tokens go to one expert.
  • Remove the crop input from the routed experts.
  • Remove guided routing.

If uncertainty routing does not beat random routing at a matched invocation rate, the gating network is not doing the work we claim.

Why not just…?

Why not just fine-tune the dense VLM longer on diplomatic data? This is a real threat. CHURRO already shows that large fine-tuning gains are possible: 14.5% on printed and 27.2% on handwritten text over base Qwen 2.5 VL. Our counter-evidence is indirect. Zhong et al. found that document OCR models are an order of magnitude more uncertain on hard spans, and that training on the hardest half of the data beats training on all of it (Reading or Reasoning?). This suggests that uniform supervision spends its capacity in the wrong place. The dense fine-tuning baseline, given the identical data budget including glyph-swap pairs, settles the question directly. If it matches our proofreader on the glyph-preservation metric, the proposal reduces to "train on glyph-swap data," and we will report it that way.

Why not just run a text-only corrector such as NeKo or a large language model? Irvine et al. show that most historical OCR errors are modernizations in which both readings are valid words. No text-only model can separate them from context alone. The glyph-swap probe, described below, makes this measurable. A corrector without the image must score at chance on swapped words, because context is held constant by construction.

Why not just re-read low-confidence lines at higher resolution with the same model? This is the simplest form of selective looking-again, and it may capture much of the gain. We expect it to fall short for one reason: the same decoder brings the same language prior, so re-reading should repeat the same modernization. We include it as a baseline, thresholded to the same fraction of re-read tokens as our router invokes. If it matches our system, then the specialist experts are unnecessary and the finding is about selective re-reading alone.

Why not just apply a dense deliberation pass everywhere? If the dense second pass matches our accuracy, the mixture-of-experts contribution shrinks to compute savings. We will then check whether those savings survive dispatch overhead in wall-clock time. Our specific prediction is that dense proofreading over-corrects tokens that were already right. We measure this as the rate at which correct draft tokens are changed into errors.

Proposed Experiments

Evaluation sets. The main evaluation uses diplomatic-transcription subsets of CHURRO-DS. A known risk is that the 155 corpora normalize differently, with some mapping ſ to s. We will audit each corpus's guidelines and report results only on corpora whose transcriptions preserve original glyphs. We add three further sets:

  • Lettres Taïtiennes, whose critical-edition ground truth gives an error taxonomy with a modernization category. Reference points are ABBYY at 35.93 errors per page, professional transcribers at 1.39, and supervised students at 0.71. No reported result was found on Lacuna for CHURRO or any vision-language model on this text.
  • Old Bailey, where Ocular's 24.1% word error rate against ABBYY's 40.0% is the reported reference (Unsupervised Transcription of Historical Documents).
  • The Chronicling Germany out-of-distribution pages, which test Fraktur under layout shift.

All of these are held out from proofreader training.

Metrics. The headline metric is normalized Levenshtein similarity. Next to it we report a glyph-preservation rate: the fraction of archaic or ambiguous ground-truth glyphs that are transcribed exactly. We also report the over-correction rate and wall-clock cost per page.

The glyph-swap probe. This is the decisive check for visual laziness. On held-out pages we splice same-font alternative glyphs into real words. We then measure how often each system's output follows the pixels rather than the expected modern word. The probe is the document analogue of Ive et al.'s masked-ambiguous-word test, where visual deliberation moved METEOR from 48.4 to 49.8.

Expected outcomes and what they mean.

  • If CHURRO already follows the pixels on swapped glyphs more than 95% of the time, the visual-laziness premise is false. The project then reduces to an efficiency study.
  • If our system raises glyph preservation while invoking routed experts on under 15% of tokens and lowering over-correction relative to the dense second pass, the claim stands.
  • A final check examines what the experts learned. For each expert, we measure the agreement between the confusion families it is routed and the guided labels on the held-out set, after guidance is removed. This shows whether the experts kept their visual specializations once free to route, or collapsed into a single generic corrector.