When a historical page is scanned, the optical character recognition (OCR) engine returns more than a string. It also gives where each glyph sat on the line, how wide its box was, how far it drifted from the baseline, and how confident the engine was. Most post-correction systems throw all of this away and rewrite the bare string with a text-to-text model. That choice causes two familiar failures. The model either copies errors through unchanged, or it "fixes" characters that were already correct. Yet many OCR errors are visibly physical. Two glyphs printed too close together get read as one ("rn" becomes "m"), an ink blot in an odd gap becomes a spurious character, and a missing space sits exactly where the gap is wide. Physical-printing models of OCR showed long ago that spacing and baseline drift carry this signal (Unsupervised Transcription of Historical Documents).
We propose to treat a recognized text line as a marked point process: a set of events at positions along the line, where each event carries a label, here the character. Post-correction then becomes a learned sequence of edits. The model inserts, deletes, or substitutes characters, and how likely each edit is depends on the measured gaps and offsets around it. Technically, this extends edit-based flow matching for point processes from unmarked timestamps to marked events. Its authors list that extension as open (Edit-Based Flow Matching for Temporal Point Processes). The edit process starts from the OCR output itself rather than from random noise. "Leave this character alone" is therefore the default, and every change has to be justified by a learned, geometry-dependent rate.
Core Claim
Our claim is that conditioning each edit's rate on the local line geometry and the engine's confidence reduces over-correction on clean text without giving up recall on noisy text. In point-process terms, the geometry is the realized gap before and after a character and its vertical offset. The probabilistic edit formulation also gives calibrated per-edit probabilities, so an archive can choose its own precision. We will also report sampling speed, but it is secondary. The burden of proof is specific. If a byte-level text model that receives the same geometry and confidence features as extra input matches us, then the point-process machinery is not earning its keep, and we will report that.
The Model
A line is represented as $X = \{(x_i, y_i, c_i, k_i)\}_{i=1}^{N}$. Here $x_i$ is the horizontal centre of glyph box $i$, $y_i$ its vertical offset from a fitted baseline, $c_i$ the engine's confidence, and $k_i$ the character. The derived quantities $\Delta x_i = x_i - x_{i-1}$ play the role that inter-arrival times play in temporal point processes.
EDITPP (Edit-Based Flow Matching for Temporal Point Processes, detailed in Lüdke et al.) generates event sequences with a continuous-time Markov chain over edit states. In plain words, the sequence is changed by random insert, delete, and substitute moves, each firing at a learned rate. Training uses discrete flow matching against Needleman-Wunsch alignments, and positions are binned. EDITPP handles timestamps only. We supply that method and add marks, so each operation also acts on the character. An insertion picks a gap bin and a character. A substitution may change the character, nudge the position, or both. Deletion stays as it is.
Following the finding that conditioning marks on realized inter-arrival time improves both likelihood and calibration (Revisiting the Mark Conditional Independence Assumption in Neural Marked Temporal Point Processes), every rate is a function of local geometry and not only of a pooled history vector:
$$u_\theta(\text{op}, i, k \mid X_s, s) = \mathrm{softplus}\big(f_\theta(\mathbf{h}_i(X_s, s),\ \log \Delta x_i,\ \log \Delta x_{i+1},\ y_i,\ c_i)\big)_{\text{op},k}.$$
The terms are defined as follows. $u_\theta$ is the rate of applying operation op at event $i$ with character $k$. $X_s$ is the current line state at flow time $s \in [0,1]$. $\mathbf{h}_i$ is a transformer embedding of event $i$ in context. $f_\theta$ is a small network. CLNM's own head is a softmax over marks, but rates must be non-negative and need not sum to one, which is why we pass $f_\theta$ through a softplus.
The source state at $s=0$ is the OCR line, and the target at $s=1$ is the ground-truth transcription. Characters in the ground truth have no boxes. We therefore place inserted characters at interpolated positions inside the aligned gap. Geometry acts mainly as a conditioning signal; it is not something we predict. This is a deliberate simplification. Our point-process innovation is narrow: edit rates conditioned on marks and gaps, plus a data-defined source. We do not claim a full spatial generative model of glyphs.
Research Program
The work has three parts.
First, we build and validate the marked edit flow on ordinary marked event data. The goal is to show that the extension is sound before OCR enters. We use the categorical mark suite (LastFM, MOOC, Github, Stack Overflow, Retweets, Reddit) used by CLNM and compare against unmarked EDITPP plus independent mark prediction. This tests whether coupling marks to gaps inside the edit process helps generation, measured by MMD and Wasserstein distance on times and mark-type distributions.
Second, we build the OCR pipeline on corpora where page images are available. We re-run Tesseract with character-level box and confidence output. The corpora are the Deutsches Textarchiv/OCR-D prints of A Two-Step Approach for Automatic OCR Post-Correction, the Swedish newspapers of Post-OCR Correction of Digitized Swedish Newspapers with ByT5, Finnish NLF Ground Truth from OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches, and Old Bailey and Trove from Ocular. The BNF corpus (OCR Error Correction Using Statistical Machine Translation) has no linked geometry, so it serves only as a text-only robustness check in which the geometry inputs are masked.
Third, we run a controlled noise-sensitivity study. Clean transcriptions are rendered with an Ocular-style typesetting and inking simulator in which baseline wander, kerning jitter, and ink density are dialed up step by step. We then OCR the renders. This gives error-rate-versus-noise curves for every system and shows whether geometry conditioning helps most where Ocular's analysis says it should: under spacing and baseline noise rather than glyph-shape noise.
Baselines
The naive reference is the uncorrected OCR output. Next to it sits a confidence-threshold rule that hands only characters below an engine-confidence cutoff to a corrector. This tests whether raw confidence alone delivers the gain.
The first strong text-only competitor is ByT5 fine-tuned per corpus, as in Post-OCR Correction of Digitized Swedish Newspapers with ByT5. That paper reports CER dropping from 3.20% to 2.04% on Swedish newspapers. It is the most direct byte-level competitor and does not depend on any tokenizer's vocabulary.
The second is the two-step detector-plus-corrector of A Two-Step Approach for Automatic OCR Post-Correction. It reports CER falling from 4.3% to 0.9% on DTA prints and altering only 0.3% of valid characters, against 6.0% for one-step seq2seq. It is the strongest existing answer to over-correction, the exact failure we target.
The third is Llama-3.1-70B with alignment-based removal of overgenerated text, plus GPT-4o where budget allows, following OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches. That paper reports up to 38.7% CER reduction on English ECCO-TCP, a 47% CER increase for Llama on Finnish, and 11.9% reduction for GPT-4o on Finnish.
The baseline we consider most dangerous is geometry-augmented ByT5. It is ByT5 given the same per-character features ($\Delta x$, $y$, $c$), quantized and interleaved as extra input tokens. The ablations are the marked edit flow with geometry inputs zeroed, which leaves a discrete text-only edit flow close in spirit to the editing actions of Levenshtein OCR, and the edit flow started from random noise instead of the OCR line.
Our own numbers come from re-run OCR, so the published figures above serve as sanity anchors only. Every baseline will be retrained on our exact OCR outputs and splits. All models get a matched parameter budget (around ByT5-base scale), matched GPU-hours, and the same number of hyperparameter trials. LLMs are run with the published prompting and alignment procedure, untuned.
Beating a baseline means lower CER and WER on the full test set. It also means a valid-character alteration rate at or below the two-step system's on the low-CER split, without the detector's lower recall. Against geometry-augmented ByT5, we require a statistically significant CER gap across the three-plus corpora, or else a clear advantage in calibration or throughput at equal accuracy.
Why not just…?
Why not just prompt a large language model? The No Free Lunches study shows this route is language-fragile. Llama-3.1-70B helps English and damages Finnish by 47%, and outputs need external alignment to strip overgenerated text (OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches). An edit process that can only modify the existing line cannot overgenerate by construction. We include LLMs on Finnish and German to confirm that this structural constraint matters.
Why not just feed box geometry and confidence into ByT5? This is the real threat. Nothing in the file shows it fails. The ByT5 authors only note that their model lacked such inputs (Post-OCR Correction of Digitized Swedish Newspapers with ByT5). Our bet is that a model that regenerates the whole line still pays an over-correction tax, while an edit process anchored at the OCR source makes copying free and makes each change local. The head-to-head with geometry-augmented ByT5 settles this, especially through valid-character alteration rates on the low-CER split. If ByT5 ties, our contribution shrinks to the marked edit-flow method and its calibration, and we will say so.
Why not just use the two-step detector with OCR confidence as an extra feature? The two-step pipeline already limits damage to valid characters, but its detector recall of 74% caps how many errors can be fixed (A Two-Step Approach for Automatic OCR Post-Correction). Its hard flagging also cannot express gap-dependent insertions such as missing spaces. Our model replaces the hard detector with soft, geometry-dependent edit rates. The confidence-threshold baseline and an augmented two-step variant isolate whether soft rates beat a hard gate.
Why not just clean the image first? Restoration models such as DocPure and DocWaveDiff are complementary rather than competing. Ocular's results show that engines fail on archaic typography and baseline wander even on legible scans (Unsupervised Transcription of Historical Documents). Our method also needs only the engine's box output, not re-processing of the images. Restored-image OCR followed by text-only correction will be one row in the noise-sensitivity study.
Proposed Experiments and Decisive Checks
The main table reports CER and WER, together with the valid-character alteration rate, on DTA/OCR-D (full and low-CER splits), Swedish newspapers, Finnish NLF, Old Bailey and Trove for every system above.
Three checks decide the proposal. The first is the gap to geometry-augmented ByT5, which tells us whether the edit-process formulation matters beyond the features themselves. The second is the gap to the geometry-zeroed edit flow, which tells us whether geometry conditioning matters within our model. We break this gap down by error category (merged glyphs, spurious insertions, segmentation errors, plain substitutions), and we predict that the gains concentrate in the first three. The third is the gap to the noise-sourced edit flow, which tests whether anchoring the process at the OCR output is what suppresses over-correction.
We will also measure expected calibration error of the per-edit probabilities, and precision-recall curves when edits are accepted only above a probability threshold. These matter in practice for archives that must not corrupt correct text.
On the synthetic renders, we plot each system's CER against baseline-wander and kerning-jitter levels. The proposal predicts a flatter curve for geometry-conditioned edits under spacing noise but not under glyph-shape noise. Seeing that asymmetry would be strong evidence for the mechanism. We report throughput in lines per second at matched accuracy. On the marked event-sequence suite, we report whether the marked edit flow matches or beats EDITPP combined with independent marks in MMD, as a check that the point-process extension holds up outside OCR.
References
- Unsupervised Transcription of Historical Documents
- Edit-Based Flow Matching for Temporal Point Processes
- Revisiting the Mark Conditional Independence Assumption in Neural Marked Temporal Point Processes
- A Two-Step Approach for Automatic OCR Post-Correction
- Post-OCR Correction of Digitized Swedish Newspapers with ByT5
- OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches
- OCR Error Correction Using Statistical Machine Translation
- Levenshtein OCR
- DocPure: Prompt-Free Unified Document Restoration via Degradation-Aware Structure-Guided Wavelet Modulation
- DocWaveDiff: A Predict-and-Refine approach for Document Image Enhancement with Wavelet U-Nets and Diffusion models