Fixing scanned historical text is a sequence-to-sequence problem: the model reads a noisy sequence from optical character recognition (OCR) and writes a clean one. It is also one where paired examples of noisy and clean text barely exist. The field works around this by training on synthetic noise. The best current recipe, from Guan and colleagues, trains a byte-level model (ByT5, a Transformer that reads and writes raw bytes instead of subword tokens) on artificial character-shape confusions. When the model still hallucinates proper nouns, it is fine-tuned again on each target document at test time. That recipe has two problems. Byte-level autoregressive decoding writes one byte per forward pass, so long documents are slow. The per-document fine-tuning step also rules out streaming use. Separately, the generative-modeling community has learned to turn many-step samplers into one-step maps. The key tool is "average velocity": a network learns the net displacement over a whole time interval instead of the instantaneous direction at one moment.
We propose to join these two lines through one observation. A model that can jump along a path in any number of steps must give the same answer for one long jump as for two short jumps chained together. That agreement can be checked on a real noisy document even when no clean version exists. We would build a one-step, non-autoregressive byte-level corrector that flows from the embedding of the corrupted input to the embedding of clean text. It is trained with supervised average-velocity targets on synthetic pairs, plus a jump-consistency loss on unpaired real OCR output. The one-step structure then does double duty: it removes the byte-by-byte latency, and it gives a self-supervised signal that pulls the model toward the real error distribution without per-document fine-tuning.
Core Claim
The claim is not simply "flow matching is faster than ByT5." A reviewer could fairly answer that with cheaper decoding tricks. The claim is that composing jumps along a learned transport from corrupted text to clean text is an adaptation signal that synthetic-only training lacks.
The setup is as follows. Let $z_1$ be the continuous byte embeddings of an input, padded with blank slots so that insertions and deletions fit. Let $z_0$ be the clean target. Let $u_\theta(z_t, r, t)$ be the network's predicted average velocity from time $t$ back to time $r$. A one-step correction is $z_0 \approx z_1 - u_\theta(z_1, 0, 1)$. On a real unpaired input, the model should agree with itself:
$$ u_\theta(z_1,0,1) \;=\; (1-r)\,u_\theta(z_1,r,1) \;+\; r\,u_\theta(\hat z_r,0,r), \qquad \hat z_r = z_1-(1-r)\,u_\theta(z_1,r,1), $$
where $r\in(0,1)$ is a randomly chosen intermediate time and $\hat z_r$ is the model's own partial repair. This needs no clean text.
On synthetic pairs, where $z_0$ is known and the path is straight, we train with the MeanFlow identity from Mean Flows for One-step Generative Modeling:
$$ u = v - (t-r)\frac{du}{dt}, $$
Here $v = z_1 - z_0$ is the instantaneous velocity of the straight path. The time derivative is computed with a Jacobian-vector product, which MeanFlow reports costs about 16% extra training compute.
The hypothesis is that the consistency term moves the model's fixed points toward the structure of real OCR errors. We expect gains on error types the synthetic generator under-represents and on proper-noun retention, which is exactly where synthetic-only ByT5 needed test-time adaptation.
Research Program
The work builds on two papers that motivate and constrain it. Advancing Post-OCR Correction: A Comparative Study of Synthetic Data shows that byte-level models trained purely on glyph-similarity noise cut CER (character error rate) substantially across Russian, English, Frisian and Telugu. That is why we keep byte inputs instead of subwords. Effective Synthetic Data and Test-Time Adaptation for OCR Correction supplies two constraints:
- Synthetic noise above about 20.1% CER hurts accuracy, its "optimal alignment curve."
- Proper nouns are hallucinated unless one runs per-document test-time adaptation (SCN-TTA).
Our consistency loss is meant to replace that per-document loop with a single model trained once on a pool of unpaired real documents. We reuse their noise generators from the NikoGuan/P_OCR codebase for the synthetic half.
The method comes from two sources:
- Flow Matching for Conditional Text Generation in a Few Sampling Steps (FlowSeq) showed that straight flows over word embeddings reach 1 to 4 network evaluations. It reports a 2,000× speedup over the multi-step text diffusion model DiffuSeq and shows that its auxiliary "anchor" cross-entropy loss (predict clean tokens from any intermediate state) prevents embedding collapse. Without it, Quasar-T BLEU fell from 0.162 to 0.001. We keep the anchor loss. We differ from FlowSeq in two ways: we start the flow from the corrupted input rather than Gaussian noise, and we target average velocity rather than instantaneous velocity. FlowSeq was trained only on high-resource clean pairs and never used unpaired data.
- MeanFlow supplies the training identity but was developed for class-conditional images. It never conditions on a source sequence or touches discrete data.
The architecture is a bidirectional Transformer over byte slots. Each input byte is followed by $k$ blank slots, with $k=1$ initially, chosen so the slots can absorb OCR insertion errors. Training targets are formed by edit-aligning the clean text into these slots. A byte classifier reads out the final state. The training mixture has three terms:
- the MeanFlow loss on synthetic pairs, with corruption levels sampled across and beyond the 20.1% threshold;
- the anchor loss;
- the consistency loss on real unpaired OCR output, with weight ramped up after the supervised terms stabilize.
The identity map trivially satisfies consistency. Only the supervised terms prevent that collapse, so we will monitor the edit rate on real inputs and stop or reweight if it drifts toward zero.
The analytical half of the program addresses an open question the file raises: does heavier synthetic noise bend the learned transport and break one-step accuracy, or does training over a range of noise levels make the model more tolerant of it? We will measure trajectory curvature, defined as the gap between one-step and 8-step outputs, as a function of the training noise ceiling. We then check whether the CER optimum still sits near 20.1% or shifts.
Grammatical error correction is a secondary transfer test, not the core. We would use Ukrainian UA-GEC with synthetic data from Low-Resource Grammatical Error Correction: Selective Data Augmentation with Round-Trip Machine Translation (SeLex-RT). SeLex-RT builds word-confusion sets from round-trip translation and reported gains of 64.1 to 66.1 F0.5 for multilingual T5. We expect this transfer to be harder because learner errors rewrite whole spans rather than single characters.
Baselines
The naive reference point is the identity: output the OCR text unchanged. All CER reductions are measured relative to it.
ByT5 trained on the same synthetic data. This is the autoregressive model of Guan et al. We retrain it on exactly our synthetic mixture at matched parameter count and training compute. It is the accuracy anchor and the latency reference.
ByT5 with SCN-TTA. This is the strongest existing method. Its 7-step pipeline masks proper nouns, builds a document-specific synthetic set and fine-tunes before inference. On RETAS it reports a 68.67% CER reduction, and it raises proper-noun Correct Word Retention Rate from 0.736 to 0.887 (Effective Synthetic Data and Test-Time Adaptation for OCR Correction). We run their released code. Its wall-clock time includes the adaptation loop. It gets the same real unpaired documents that our consistency loss sees, so neither method has privileged access to target-domain text.
FlowSeq. This baseline uses a Gaussian prior with the source concatenated as context. It is trained on the same pairs at 1 and 4 network evaluations and isolates our choice to start the flow from the corrupted input. No reported result was found on Lacuna for FlowSeq on any OCR or error-correction benchmark.
Key ablations. First, the same network trained only on direct 1→0 jumps with no consistency term. This tests whether jump composition matters beyond simple non-autoregressive regression. Second, the full model with the consistency term replaced by pseudo-label self-training on the same real documents.
What counts as beating the baselines. We need RETAS CER reduction within one point of 68.67% and proper-noun retention at or above 0.887. Both must hold without per-document fine-tuning, at a measured order-of-magnitude lower wall-clock time per document than ByT5 on the same GPU. We also need the consistency ablation to show a statistically reliable gap over both the direct-jump and pseudo-label variants.
Why not just…?
Why not just keep autoregressive ByT5 and speed it up by using the input as a draft? OCR output is mostly correct, so copying input bytes and verifying them in parallel could recover much of the latency while preserving ByT5's exact accuracy. No reported result was found on Lacuna for draft-and-verify decoding on post-OCR text, but this is a real threat to any pure-speed claim. That is why speed is not our core claim. We will implement this baseline and report latency against document CER, because acceptance rates should fall as noise rises. If it matches our latency at every noise level, the remaining contribution must come from the consistency signal removing SCN-TTA. We would then reframe the work around adaptation rather than speed.
Why not just self-train ByT5 on pseudo-labels of the real documents? Pseudo-labeling uses the same unpaired data and is simpler. Its weakness is that it reinforces the model's own mistakes, including confident proper-noun rewrites. That is exactly the failure SCN-TTA was built to fix with masking, which suggests naive self-training does not solve it. The consistency loss, by contrast, only demands agreement between repair routes and never treats an output as ground truth. The pseudo-label ablation above settles this directly. If it matches the consistency variant, our mechanism is not doing the work.
Why not just use a single-pass edit tagger? Taggers in the style of GECToR are already one-step. They rely on subword tokenizers and fixed edit vocabularies, though, and the file's evidence is that subword tokenization breaks under heavy OCR noise, which is why the byte-level models won. Our direct-jump ablation is effectively a byte-level one-step regressor. If it matches the full model, the flow structure adds nothing over a tagger-like design and we would say so.
Why not just prompt a large language model? Guan et al. report that synthetic-trained ByT5 outperformed GPT-4o and Llama 2 in one-shot historical OCR correction. Per-document cost at that scale is also far above a compact corrector, so we include this only as a reference row.
Proposed Experiments
The primary evaluation uses RETAS, ICDAR2017, ICDAR2019 and BLN600, all historical OCR text with human transcriptions. We add the multilingual synthetic-to-real setting with Frisian and Telugu to probe low-resource scripts. Metrics are CER reduction, proper-noun Correct Word Retention Rate, wall-clock time per document and bytes per second on a single fixed GPU, and the number of network evaluations. The secondary test is UA-GEC F0.5, with the SeLex-RT mT5 figure of 66.1 as the external reference.
Four decisive checks follow.
Accuracy at matched latency. At equal latency, does the one-step model match or beat ByT5 with SCN-TTA on CER and proper-noun retention? If it loses badly at one step, the method fails regardless of speed.
Where the consistency gains land. Removing the consistency term should hurt most on error categories whose real frequency exceeds their synthetic frequency, computed from aligned edits. If its benefit is uniform across categories, or zero, the adaptation hypothesis is wrong.
Scaling with unpaired data. We vary the number of real unpaired documents from zero upward. A gain that grows with this pool, while the pseudo-label variant plateaus or degrades, is the result that would justify the approach.
Curvature against noise ceiling. We sweep the training noise ceiling from 10% to 40% CER and track one-step versus multi-step disagreement alongside test CER. This tells us whether one-step transport inherits the 20.1% ceiling, moves it, or breaks at high noise. Each outcome would be a publishable characterization of an open question.
References
- Mean Flows for One-step Generative Modeling
- Advancing Post-OCR Correction: A Comparative Study of Synthetic Data
- Effective Synthetic Data and Test-Time Adaptation for OCR Correction
- Flow Matching for Conditional Text Generation in a Few Sampling Steps
- Low-Resource Grammatical Error Correction: Selective Data Augmentation with Round-Trip Machine Translation