Diversify the Reading, Not the Reader: Ensembles Over OCR Transcripts for Faithful Answers from Scanned Documents

When a language model answers questions about a scanned document, it does not see the page. It sees what an OCR system (optical character recognition, the software that turns a page image into text) produced. Where that text is garbled, the model fills the gap with whatever its training makes plausible, and the result is a confident, fluent, wrong answer. A natural fix is to ask several small models and take a vote, since ensembles of small models have recently matched much larger single models on clean reasoning tasks. This proposal argues that on scanned documents such ensembles mostly fail, and for a simple reason. Every model reads the same corrupted characters and fills the same gap with the same plausible guess, so their errors move together and voting cannot cancel them. The fix is to diversify the input instead of the models: run several independent OCR and restoration pipelines, have a small model answer from each transcript separately, and treat the answers' agreement, and the places where the transcripts disagree, as the evidence for answering or abstaining.

In two sentences: OCR noise makes model ensembles fail together because the noise is shared, so we ensemble over independent readings of the page rather than over readers. Agreement across readings decides the answer, and disagreement that sits exactly on the answer's supporting text tells the system to abstain rather than guess.

Core Claim

The claim has three testable parts. The first is that on OCR-degraded inputs, the errors of different language models become strongly correlated as noise grows, so model-diverse ensembles gain little over their best member. The second is that an ensemble with the same number of model calls, spread over transcripts from different OCR engines and image-restoration steps instead of over different models, recovers a large share of the accuracy lost to OCR noise. The third is that disagreement across transcripts, localized to the text that supports a candidate answer, is a better trigger for abstention than self-consistency or internal-uncertainty signals from a single model.

The reasoning behind the first two parts is the standard variance formula for averaging correlated predictors. If $K$ ensemble members each have error variance $\sigma^2$ and pairwise error correlation $\rho$, the averaged error variance is

$$\mathrm{Var}_{\text{ens}} = \rho\,\sigma^2 + \frac{(1-\rho)\,\sigma^2}{K},$$

so adding members only removes the $(1-\rho)$ part. Shared OCR corruption pushes $\rho$ toward one for model-diverse ensembles. Independent OCR engines fail on different characters, which lowers $\rho$ at the source. We treat this as a lens, not a theorem about discrete answers. It tells us what to measure, namely $\rho$ as a function of noise level for each kind of ensemble.

Where This Sits in Prior Work

Three papers motivate the gap. OCR Hinders RAG built OHRBench and showed that moving from mild to severe semantic OCR noise cuts retrieval-augmented question answering by nearly 50%. Retrieval-augmented generation (RAG) means answering from passages fetched out of a document store. That paper also found that even the best OCR leaves at least a 14% F1 gap to ground-truth text, and that only scale helped: Qwen2-72B answered 9.4% and 3.5% more questions correctly under formatting and semantic noise than smaller models. It tests single models only and offers no mechanism beyond "use a bigger model." Seeing is Believing? identifies the mechanism we target: when visual evidence is degraded, models fall back on language priors and invent plausible text. It fixes this by training one 7B vision-language model with reinforcement learning to tag or refuse illegible fields. We pursue the same refusal behavior without training, using cross-transcript disagreement, and we study text-only pipelines over standard OCR output. Its fine-tuned model is a strong comparison point on its own benchmark. Finally, the clinical allergy-extraction system of Berge et al. found that OCR failures caused 32.3% of extraction errors on scanned hospital records, which shows the problem is practical.

Two ensemble papers supply our method and also constrain it. SLM-MUX showed that debate among small models causes "groupthink" and lowers accuracy by over 5%. It instead has each model answer in isolation and aggregates with a non-LLM rule based on self-consistency and validation accuracy. It names the "consistently wrong" failure, where models sharing a bias agree confidently on a wrong answer. We reuse its isolation and non-LLM aggregation, and we argue that OCR noise is exactly the condition that produces its named failure. MAC masks the agent least consistent with the group as a proxy for hallucination. That rule assumes the majority is right, which shared input noise breaks. Both were evaluated only on clean benchmarks. The divide-and-conquer noise decomposition splits system error into task, model, and aggregation noise, but it has no term for corruption of the input itself. Our analysis adds that term and separates it into a shared part and a view-specific part.

On the hallucination side, noise-injection sampling perturbs a single model's internal activations to estimate its parameter uncertainty, raising detection AUROC on GSM8K from 76.53 to 82.70 for Llama-3.2-3B-Instruct. It measures what the model does not know. It cannot see that the input itself is wrong, which is the failure we target, so it serves as a baseline. HaluAgent verifies claims with external tools, and HaluEval shows that external knowledge raises QA hallucination detection from 62.59% to 76.83%. Our "external evidence" is a second reading of the same page, a check that requires no search engine.

Research Program

The system has three stages. In the reading stage, each document page is processed offline by $V$ independent pipelines. These are a classical pipeline OCR engine, an end-to-end OCR model, and a vision-language model used as a transcriber, matching the three engine families OHRBench already evaluates. Each is run on the raw image and on an image cleaned by a learned binarizer such as DocEnTr, a transformer that restores degraded page images (19.11 PSNR and 92.53 F-measure on DIBCO 2017). This yields up to six transcripts per page. They cost a one-time indexing pass, not per-query compute.

In the answering stage, a single small model answers each query separately from the passages retrieved within each transcript. No transcript sees another, and no model sees another model's answer, following the SLM-MUX finding against debate.

In the aggregation stage, a deterministic rule replaces voting by an LLM. Candidate answers are clustered by normalized string match. Each cluster is scored by how many independent views produced it, with views from the same engine family discounted. Each cluster must also pass a provenance check: the answer string, or a fuzzy match within a small edit distance, must appear in the retrieved text of at least $m$ views. An answer that every view produces but that no view's text contains is flagged as filled in from priors. When the top clusters split across views and the transcripts differ on the characters of the supporting span, the system abstains. Disagreement on spans elsewhere on the page does not trigger abstention. The thresholds $m$ and the abstention margin are tuned on a held-out split.

The analysis program measures pairwise error correlation $\rho$ among models, and separately among views, at each OHRBench noise level. It then fits the variance relation above to check whether $\rho$ predicts the observed ensemble gain. This tests the explanation, not only the accuracy.

Baselines

The naive reference point is one small model, such as Qwen2.5-7B, answering from the single best OCR transcript, with ground-truth text as the upper bound. The scale reference is Qwen2-72B on the best transcript. OCR Hinders RAG reports it as the most noise-robust model tested, with 9.4% and 3.5% more correct answers than smaller models under formatting and semantic noise. Our ensemble must match it at lower total compute.

The strongest existing ensemble competitor is SLM-MUX run with its own offline team selection over a pool of 7B to 24B models, all reading the single best transcript. It is the strongest because it already handles groupthink and confident contradictions. SLM-MUX reports two small models matching or beating Qwen 2.5 72B on MATH, GPQA, and GSM8K. No reported result was found on Lacuna for SLM-MUX on OHRBench or any OCR-noisy input, so we run it ourselves. We also run a MAC-style consistency mask on the same pool.

The uncertainty baselines are self-consistency with $K$ temperature samples from one model on one transcript, and noise-injected sampling from Enhancing Hallucination Detection through Noise Injection. These are the strongest single-model abstention signals in the file.

The repair baselines first correct the text and then answer once. One fuses the $V$ transcripts by character-level alignment and voting. The other applies a post-OCR correction model in the spirit of chained SMT correction, which cut WER from 4.9% to 1.9% on historical French, or a small LLM prompted to correct the text.

On KIE-HVQA, the reference is the RL-trained Qwen2.5-VL-7B of Seeing is Believing?, which reports a 58.05% Average Distance score against GPT-4o's 30.21%. We compare on that metric using the released test split. If the weights are available we re-run them; otherwise we compare against the reported figure.

Fairness rules apply to all methods. Every method gets the same retriever, the same document set, and a matched budget of model calls per query: $K$ calls spread over models, samples, or views. Every method also gets the same tuning split for thresholds. The ablation that isolates the key ingredient is a two-by-two design that crosses model diversity (one model versus several) with view diversity (one transcript versus several) at equal call counts, plus a version with the provenance check removed. Beating the strongest competitor means a statistically significant gain in end-to-end F1 over SLM-MUX at severe semantic noise. On selective prediction, it means a better risk-coverage curve than self-consistency and noise injection, with no loss on clean ground-truth text.

Why not just…?

Why not just use a bigger model? OHRBench shows scale helps, but by single-digit percentages against a roughly 50% collapse under severe noise. A 72B model reading "1,8O0" still has to guess. Our design includes Qwen2-72B on the best transcript. We also run Qwen2-72B on multiple views, which tests whether view diversity helps large models too. If the single large model matches the small view ensemble at similar cost, the practical case weakens, but the abstention claim can still hold.

Why not just fuse the OCR outputs into one clean transcript first? This is the most serious alternative, and we treat it as a real threat. Character-level voting is cheap and old. It fails exactly where engines split without a majority, because it must commit to one string without knowing which characters matter for the question. It then discards the disagreement that our abstention rule relies on. The fusion-then-answer baseline settles this directly. If it matches our accuracy, the contribution narrows to the abstention signal, and the risk-coverage comparison decides whether that survives.

Why not just correct the OCR text with a model before answering? The research file notes that correction systematically rewrites rare names, archaic spellings, and domain terms into frequent words (OCR Error Correction Using Statistical Machine Translation). That is the same prior-driven filling-in that causes hallucination, moved one step upstream. We expect correction to help on common words and to hurt on entity and number questions, and we report results split by answer type to check this.

Why not just use self-consistency or internal-noise uncertainty on one model? Both measure how unsure the model is given its input. When the input says something wrong but readable, the model can be consistently and confidently wrong. Repeated samples share the same corrupted context, so their correlation is near one by construction. The correlation measurements and the abstention AUROC comparison test this directly.

Proposed Experiments

The main experiment runs every method on OHRBench across its seven domains and its mild-to-severe semantic and formatting noise levels. We report end-to-end F1, accuracy split by answer type (number, entity, free text), risk-coverage curves, abstention AUROC, error correlation $\rho$ for model-diverse and view-diverse ensembles, and calls and wall-clock time per query. The second experiment uses KIE-HVQA. We run the same OCR engines on its degraded images and score with its Average Distance metric. This tests whether training-free cross-reading abstention approaches the RL-trained refusal model. The third experiment is a small transfer check on handwritten notes from GNHK, one of the collections OHRBench draws on, where engine disagreement should be highest.

Three outcomes would decide the question. The claim is supported if, in the two-by-two design, view diversity yields a larger gain than model diversity at severe noise and a smaller gap on clean text, and if measured $\rho$ explains that difference. The claim is falsified if model-diverse SLM-MUX gains as much as view diversity under severe noise, since that would mean shared input noise does not drive ensemble failure. Removing the provenance check should mainly raise errors on number and entity questions where all views agree but the answer appears in none of them. If it does not, that component is unnecessary and we would drop it.