Systems that pull structured facts out of scanned documents fail when the text-recognition step garbles characters. In one hospital deployment, recognition errors on scanned records caused 32.3% of all allergy-extraction failures (Combining Unsupervised, Supervised, and Rule-based Algorithms for Text Mining of Electronic Health Records). The standard fix is to make the model's text representation invariant to character noise: inject random typos during training and pull each corrupted sentence toward its clean version. That is cheap and it works for retrieval. It has a built-in hazard, though. Random typos often turn one real word into a different real word ("Celebrex" into "Cerebyx", "5 mg" into "50 mg"). Uniform invariance then teaches the model to merge exactly the distinctions a clinical or financial extractor must keep.
We propose to learn the invariance from the task instead of from the noise. We cast document extraction as a sequence of decisions and train the document encoder with a bisimulation metric. Bisimulation is a notion from reinforcement learning (RL): two states count as close only if they yield similar rewards now and lead to similar states afterwards. Under this metric, an OCR error that does not change what the extractor should do, now or later, costs nothing and is collapsed. An error that turns a fact into a different fact keeps its distance. In two sentences: bisimulation gives a graded, task-defined answer to "which OCR differences matter," where noise augmentation answers "none of them." We test whether that answer improves extraction on degraded scans without losing look-alike distinctions.
Core Claim
Behavioral equivalence is the right invariance target for OCR-robust extraction. The bisimulation metric was developed for visual control. There, Deep Bisimulation for Control (Learning Invariant Representations for Reinforcement Learning without Reconstruction) showed that grouping observations by reward and transition behavior discards moving-video backgrounds that reconstruction-based methods waste capacity on.
OCR noise is the textual analogue of those backgrounds, with one important difference. Some of it is not irrelevant. A character substitution in a drug name, a dosage, a negation, or an invoice total can change the correct output. The metric's reward term protects these cases. Its recursive transition term adds something reward-only supervision lacks: a corruption that looks harmless at the current step but derails a later decision is also kept apart.
The concrete prediction has two parts. First, a bisimulation-trained extractor should match noise-augmented encoders on ordinary OCR degradation. Second, it should clearly beat them on documents containing confusable entities, meaning distinct valid values within one or two character edits of each other.
Research Program
The decision process follows ITERX (Iterative Document-level Information Extraction via Imitation Learning). ITERX extracts one multi-slot template per step, tracks what has already been extracted with a recurrent state, and trains by imitating a dynamic oracle on clean text. We keep its actions and oracle but feed it OCR output. The state $s$ is the encoded document plus the extraction history so far. The reward $r$ is the gain in template-matching score against gold produced by each step.
Because training uses fixed oracle and policy rollouts rather than live interaction, this is an offline setting. That makes the failure mode identified in Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning a direct constraint: with incomplete coverage, the metric's training loss can reach zero while distances stay wrong. We therefore adopt that paper's expectile operator (an asymmetric squared loss) with $\tau=0.6$, and its reward rescaling by $(1-\gamma)$, as defaults rather than optional extras.
The encoder $\phi$ is trained with an auxiliary loss over pairs of states $(s_i, s_j)$ drawn from the batch, in the style of DBC:
$$\mathcal{L}(\phi)=\Big(\lVert \phi(s_i)-\phi(s_j)\rVert_1-(1-\gamma)\,|r_i-r_j|-\gamma\, W_2\big(\hat P(\cdot\mid \phi(s_i),a_i),\hat P(\cdot\mid \phi(s_j),a_j)\big)\Big)^2$$
Here $a_i$ is the action taken in $s_i$, $\gamma$ is the discount factor, and $\hat P$ is a learned Gaussian model of the next latent state. $W_2$ is the 2-Wasserstein distance, which has a closed form between Gaussians. The expectile weighting replaces the plain square in the offline setting.
Pairs are formed in two ways. Some pair a clean rendering with its OCR'd rendering of the same document. Others pair different documents at the same extraction step. Only the second kind can teach the metric that two clean but confusable values are far apart.
DBC's known $O(B^2)$ pair cost is handled by restricting pairs to states at the same step and slot. This also reduces the problem of comparing states whose actions are incomparable.
The same learned distance is also a diagnostic. The distance between a document's clean state and its OCR'd state is a task-aware measure of OCR damage. We will test whether it predicts extraction failure better than character error rate (CER), which weights all characters equally. A useful side effect follows from DBC's Block MDP assumption, which requires that each observation identify the underlying state. When OCR destroys information, that assumption breaks, and the metric should keep the damaged state far from the clean one rather than hide the damage.
Baselines
The naive reference points are ITERX trained on clean text and evaluated on OCR output, and ITERX trained directly on OCR output with no invariance objective. Together they measure how much noise hurts and how much plain exposure to it helps.
The strongest text-side competitor is the contrastive character-noise recipe of Cheap Character Noise for OCR-Robust Multilingual Embeddings. That method fine-tunes an encoder with 5% random character edits so corrupted text maps onto clean text. It reports a gain of more than 4 retrieval points on Blackletter text with no clean-text loss, in about 20 minutes on one GPU. We will plug it in as the ITERX encoder, using the same backbone, the same documents, and the same augmentation budget as our method. Each method gets an equal hyperparameter-search budget.
The strongest layout-side competitor is FS-DAG (FS-DAG), a spatial graph network over text boxes. It reports 93.90% F1 on WildReceipt against LayoutLMv3's 87.14%, and loses less than 1% F1 under 10% injected OCR errors. On WildReceipt, beating it means exceeding 93.90 F1 on clean data or showing a smaller drop under real OCR noise. Matching it while also winning on the confusable-entity subset, which FS-DAG was not evaluated on, would still be informative.
A pipeline baseline corrects the text before extraction with a byte-level model, ByT5. In Sanskrit it cut word error rate (WER) from 30.23% to 23.19% (A Benchmark and Dataset for Post-OCR text correction in Sanskrit). We also run image enhancement with DocEnTr (DocEnTr) before Tesseract.
Two ablations isolate the key ingredient. Setting $\gamma=0$ reduces the metric to reward differences only, which amounts to label-supervised metric learning. Removing the cross-document pairs leaves only clean-versus-noisy pairs, which reduces our method to a learned version of noise invariance. The proposal succeeds only if the full model beats both ablations on the confusable-entity subset.
Why not just…?
Why not just fine-tune with cheap character noise? On ordinary degradation it may tie with our method, and we expect that. Its objective pulls every corrupted string toward its source. When a random edit lands on another valid entity, it pulls two real, distinct values together. The confusable-entity evaluation measures exactly this. If character-noise training shows no loss there, our core claim is wrong and the paper reports that.
Why not just filter augmentations with a dictionary, so that noise that produces another valid word is discarded? This is the most serious threat, and we include it as a baseline. It works where a closed vocabulary exists, such as drug lists. It cannot judge corruptions in open-vocabulary fields, numbers, units, or context words like negations, whose importance depends on the downstream decision. If the dictionary filter matches our method on drug names, we expect our gains to show up only on numeric and context-word confusions. The per-category breakdown will show whether that holds.
Why not just correct the text first with ByT5, or clean the image with DocEnTr? The Sanskrit study found that 71% of ByT5's corrected words remained morphologically invalid. Correction also optimizes string fidelity, not the extraction outcome. DocEnTr optimizes pixel quality (19.11 PSNR on DIBCO 2017) at heavy memory cost and has no notion of which characters matter. Both run as pipeline baselines under the same evaluation, and the gap on confusable entities will show whether string or pixel fidelity is enough.
Why not just use the reward signal directly, with plain supervised learning? That is the $\gamma=0$ ablation. The recursive term earns its place only if it helps on errors whose effect appears at a later extraction step.
Proposed Experiments
We render MUC-4 and SCIREX documents to page images, apply degradations of the kinds DIBCO evaluates (fading, bleed-through, low resolution), and run Tesseract. This gives paired clean and OCR text with gold templates. ITERX's clean scores of 53.0 F1 on MUC-4 and 16.9 F1 on SCIREX are the clean ceiling. No reported OCR-noise result was found on Lacuna for either benchmark.
WildReceipt provides real scans and the FS-DAG comparison. FinCriticalED, a benchmark for fact-level financial OCR, supplies numeric confusables; no reported result was found on Lacuna for it.
Across all of these, we build the confusable-entity subset automatically. It contains test documents whose gold values have another valid value within edit distance two. Metrics are template F1 overall, F1 on that subset, and the F1 drop from clean to OCR input.
The decisive check is that bisimulation-trained ITERX loses no more than one F1 point to character-noise training on the full OCR test sets and wins clearly on the confusable subset. It must also beat the $\gamma=0$ and same-document-only ablations, and the dictionary-filtered augmentation on the non-drug categories. A secondary check concerns the diagnostic: the learned clean-to-OCR distance should rank documents by extraction failure better than CER, measured by area under the ROC curve on held-out pages. If the subset gap vanishes, the honest conclusion is that uniform noise invariance is already enough for extraction, and the bisimulation machinery is not needed.
References
- Combining Unsupervised, Supervised, and Rule-based Algorithms for Text Mining of Electronic Health Records - A Clinical Decision Support System for Identifying and Classifying Allergies of Concern for Anesthesia During Surgery
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Iterative Document-level Information Extraction via Imitation Learning
- Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning
- Cheap Character Noise for OCR-Robust Multilingual Embeddings
- FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding
- A Benchmark and Dataset for Post-OCR text correction in Sanskrit
- DocEnTr: An End-to-End Document Image Enhancement Transformer