Who Checks the Checker When the Page Is Smudged? Claim-Localized Re-Reading for LLM Judges on Scanned Expert Documents

Many teams now grade the factual accuracy of language-model answers with another language model, a setup called "LLM-as-a-judge." In finance, medicine and law, the evidence the judge checks against is often a scanned document. It reaches the judge as text from optical character recognition (OCR, the software that turns page images into characters). That text is noisy, and in expert documents the noise tends to hit exactly the characters that carry meaning: the parentheses that mark a negative number in a financial statement, a decimal point in a drug dose, a digit in a date. When one of those characters is misread, the judge can flag a correct answer as a hallucination or accept a fabricated one. Nobody currently measures how often this happens.

We propose to measure it, and to test a specific and cheap fix. The judge breaks each answer into atomic claims and finds the evidence span each claim depends on. When that span contains characters that are both meaning-bearing in the domain and unstable across OCR readings, the judge crops the image at that location and asks a vision-language model one closed question, such as whether a particular cell is enclosed in parentheses. The judge then returns one of three verdicts: supported, contradicted, or unreadable. Our bet is that most noise-induced judge errors sit in a few such characters, so fixing them locally should recover most of the lost accuracy at a small fraction of the cost of page-level fixes.

Core Claim

Our central hypothesis is that noise-induced judge errors on scanned expert documents are concentrated, not diffuse. Three conditions tend to hold together. The error sits inside the evidence span a specific claim depends on. It involves a small, domain-typed class of characters: signs, decimals, digits, units, dates and negations. And independent readings disagree on it. If this holds, judge reliability is limited by a local re-reading problem, not by global transcription quality. In that case a claim-triggered, crop-level re-read with an explicit "unreadable" verdict should beat three page-level remedies: better OCR, giving the judge the raw image, and consensus across multiple readers. If the errors turn out to be diffuse, the method fails, and we will say so. The measurement study of judge failure under OCR noise would then be the main contribution.

We summarize the result as a gap-recovery ratio:

$$R = \frac{A_{\text{method}} - A_{\text{noisy}}}{A_{\text{clean}} - A_{\text{noisy}}}$$

Here $A_{\text{clean}}$ is the judge's verdict accuracy when given the clean ground-truth document text. $A_{\text{noisy}}$ is its accuracy when given raw OCR output. $A_{\text{method}}$ is its accuracy under a given remedy. All accuracies are measured against labels defined on the clean document. $R$ near 1 means the remedy removes the damage OCR does to the judge.

Where This Sits in Prior Work

Three papers motivate the gap. FinCriticalED benchmarks OCR systems and vision-language models on 739 expert-annotated numerical and temporal facts in SEC filings. It shows that small character errors barely move edit distance but can invert financial meaning. Its scoring relies on a GPT-4o judge whose own error rate is never measured. We take its "small characters, large meaning" finding and ask what happens to the judge itself. FinGround decomposes financial answers into typed atomic claims and verifies them with recomputation and a distilled 8B classifier. We reuse its claim decomposition and taxonomy. However, FinGround assumes clean digital text, and it reports that recall falls to 71.4% on near-miss numbers within ±5% of the truth. Those near-miss cases are exactly the ones OCR noise creates. Seeing is Believing? shows that multimodal models guess plausible text when characters are occluded. It trains a 7B model with a three-state reward (legible, partially obscured, unrecognizable) so the model refuses rather than guesses. We borrow the idea that "unreadable" is a legitimate output, but we apply it to the judge's verdict rather than to transcription.

Two papers constrain the design. Consensus Entropy verifies OCR by measuring agreement among several vision-language models in a semantic embedding space. It beats single-model VLM judges by 15.2% F1 on its RealData set. However, its authors note that general embeddings miss meaningful differences in technical domains, and Gyllensten et al. show that OCR noise itself distorts embeddings. For these reasons we measure disagreement at the character level, and only inside claim-relevant spans, not in embedding space across whole pages. MLLM-as-a-Judge reports two relevant effects. Multimodal judges often do better with a text description of an image than with raw pixels, and chain-of-thought reasoning can make them invent visual details. We therefore keep the image-reading step narrow and closed-form (answer yes/no or a value, with no free-form reasoning), and we test the chain-of-thought effect directly rather than assuming it away.

Research Program

The first component is a controlled testbed built on documents with clean ground truth. FinCriticalED supplies 500 page images paired with clean HTML. For each of its 739 facts we write claim-bearing answers of four kinds: correct; a near-miss numeric error; a sign or parenthesis flip; and a date shift. The label of each answer is defined against the clean HTML. We add a clinical slice from the KIE-HVQA test split, 400 degraded invoices, identity cards and prescriptions, where dose and date fields play the same meaning-bearing role. For real rather than synthetic noise we use the Consensus Entropy RealData pages, restricted to pages we can annotate at the claim level. Each page is transcribed three ways: by an open OCR engine, by a strong document parser, and by a vision-language model. The same pages are also degraded synthetically (blur, low contrast, occlusion, in the style of KIE-HVQA). This lets us vary the noise level while holding the content fixed.

The second component is a diagnostic study. For every judge error that appears only under noisy evidence, we record three things: whether the corrupted characters lie inside the evidence span of the claim being judged, which character class they belong to, and whether the three transcriptions disagree there. Together these measurements test the concentration hypothesis before any method is built.

The third component is the judge itself, which works in three steps. First, it follows FinGround and decomposes each answer into atomic claims, then aligns each claim to an evidence span in the transcript. Second, it flags a span as fragile if two conditions both hold: the span contains characters from a domain-typed list (signs, parentheses, decimals, digits, units, dates, negation words), and the transcriptions disagree on those characters or the recognizer is uncertain about them. Third, for fragile spans only, it crops the image around the span's bounding box and asks a vision-language model a closed question generated from the claim type. If that re-read is itself uncertain, the verdict becomes "unreadable" instead of a forced "supported" or "contradicted."

Baselines

The naive reference point is the FinCriticalED-style judge used as is. GPT-4o receives a single open-source OCR transcript as evidence, uses the same normalization prompt, and must return a binary verdict. We report two versions: the clean-evidence judge ($A_{\text{clean}}$), which sets the ceiling, and the noisy-evidence judge ($A_{\text{noisy}}$), which sets the floor. For grounding, FinCriticalED reports 93.61% Financial Fact Accuracy for GPT-5 and 74.55% for MinerU2.5 on its benchmark. Those numbers measure extraction, not judging. We found no reported result on Lacuna for judge verdict accuracy under noisy OCR evidence, so establishing that number is part of the work.

The strongest text-side competitor is FinGround-style atomic claim verification run directly on the noisy transcript. It reports 91.4% F1 on FinHalu, which uses clean text, and 71.4% recall on near-miss numbers. It is the right comparison because it already handles domain claim types well; the open question is whether it degrades once OCR noise enters. Our method contains FinGround's decomposition as its first step, so this comparison isolates what localized re-reading adds.

The strongest remedy on the reading side is Consensus Entropy applied to the three transcriptions before judging. It reports 15.2% higher F1 than VLM-as-judge verification on RealData. We use its published recipe, with high-disagreement pages sent to the same frontier model we use elsewhere. We also run two further single-change remedies. One restores the image with PreP-OCR (PreP-OCR) and then transcribes it. The other gives the judge the full page image together with the transcript.

Fairness rules apply to every method. All use the same judge backbone and the same answer sets. All get the same prompt-tuning budget on a held-out development split. We report cost as model calls and image tokens per claim, alongside accuracy. To count as beating a baseline, a method must achieve higher $R$ and a lower false-accusation rate (correct answers marked "contradicted") at equal or lower cost than that baseline.

Three ablations isolate the key ingredients. The first re-reads every claim span, which tests whether the fragility trigger matters or whether re-reading everything would do. The second replaces the domain-typed trigger with generic transcription disagreement, which tests whether domain character classes matter. The third forces a binary verdict, which tests whether the "unreadable" option matters.

Why not just…?

Why not just use a better OCR engine or restore the image first? FinCriticalED shows that the system with the best edit distance can have the lowest fact accuracy. A cleaner page on average does not guarantee that the one parenthesis a claim depends on is right. Berge et al. find that recognition errors still cause roughly a third of extraction failures in clinical scans. That said, this is a real threat on lightly degraded pages. The PreP-OCR and best-parser baselines will show directly how much of the gap page-level cleanup closes, and we expect it to close more at low noise than at high noise.

Why not just give the judge the page image? This is the most serious alternative. A frontier multimodal judge that sees both the image and the transcript might already check the right cell. MLLM-as-a-Judge offers some counter-evidence: judges do better with text descriptions than with pixels, and they invent visual details under chain-of-thought. Seeing is Believing? finds that GPT-4o reaches only 30.21% Average Distance on degraded documents, meaning its free reading of the page is unreliable. Still, the full-image-plus-transcript baseline decides this question. If it matches our method's $R$ at similar cost, then localization adds nothing and we will report that.

Why not just use multi-model consensus? Consensus Entropy is strong on generic OCR. However, it measures agreement in an embedding space that its own authors say misses domain distinctions. Readers that share linguistic habits can also agree on the same wrong guess. Our character-level, claim-scoped disagreement signal is a narrower version of the same intuition. The head-to-head comparison, and the second ablation (generic disagreement trigger), show whether the narrowing helps.

Why not just sample the judge several times and look at its uncertainty? Evidence from Enhancing Hallucination Detection through Noise Injection suggests that temperature sampling mainly captures randomness in the model's outputs, not uncertainty about what the evidence actually says. Their reported AUROC on GSM8K rises from 76.53 to 82.70 only after they perturb internal activations. In our setting, a judge that confidently trusts a corrupted transcript will be consistently wrong across samples. We include a sampling-based judge given the same number of model calls as our method, to check this directly.

Decisive Checks

The first check is the concentration diagnostic. If only a minority of noise-induced judge errors fall on domain-typed characters inside claim spans, the method's premise fails. The paper would then become a measurement study of how OCR noise corrupts domain judges, including breakdowns by noise level, domain and claim type.

The second check is near-miss behavior. On sign flips and ±5% numeric errors built over noisy evidence, we compare near-miss recall and false-accusation rate between our method and FinGround-style verification. This is the regime where FinGround is already weak on clean text, and it is where we expect the largest difference.

The third check is the three-way verdict under heavy occlusion. On KIE-HVQA-style occluded fields, a judge that returns "unreadable" should agree with the clean-document label whenever it does commit to a verdict. It should also mark as unreadable the fields that Seeing is Believing? classes as unrecognizable, rather than guessing.

The fourth check repeats the re-read step with and without chain-of-thought. This tests whether the reasoning-induced visual hallucination reported by MLLM-as-a-Judge also appears in narrow, crop-level questions.

Finally, all results are reported separately for the financial, clinical and real-scan slices. This matters because a method that only works on SEC tables would not support the claim that the fix transfers across expert domains.