Was It Unreadable or Mislocated? Logit-Based Error Attribution for Question Answering on Degraded Documents

When a vision-language model (a model that reads an image and writes text) answers a question about a scanned document and gets it wrong, the error usually has one of two causes. Either the model could not read the relevant text, because the scan is blurred, stained or faded, or it read fine but looked in the wrong place: it pulled a value from the wrong table row or the wrong form field. The two failures call for different fixes. An unreadable region needs a rescan, image restoration or a human. A mislocated answer needs the model to ground its answer again. Current confidence scores merge both causes into one number, and spatial grounding failures are known to be common and invisible. On BBox-DocVQA, GPT-5 answers 81.45% of questions correctly but points to the supporting evidence with a mean intersection-over-union of only 0.90% (BBox DocVQA).

We propose a confidence score that needs no sampling and no retraining, and that splits a document answer's uncertainty into a reading part and a locating part. It uses only the model's own token probabilities over three short passes. In the first, the model answers and names an evidence box. In the second, we re-score that same answer given only the image crop inside the box. In the third, the model transcribes the crop. If the model's answer does not survive being shown only its claimed evidence, it probably answered from global cues rather than from that region ("right answer, wrong reason"). If the answer does survive but the transcription of the crop is low-probability, the problem is reading. We test whether this split predicts errors better than a single answer-level score. We also test whether it predicts which intervention fixes each error, using synthetic degradations, where the clean page and the true evidence box are both known.

Core Claim

Our claim is that error detection on degraded documents improves when the confidence signal is anchored to the model's own claimed evidence region rather than computed over the answer alone. We further claim that the gap between "answer probability given the whole page" and "answer probability given the claimed crop" is a cheap logit-based detector of spatial grounding failure. The closest work does not do either. Teaching VLMs to Admit Uncertainty in OCR from Lossy Visual Inputs trains a model with supervised fine-tuning and reinforcement learning to wrap misread words in explicit tags. It addresses only transcription, never inspects logits, and never localizes. Reading or Reasoning? Format Decoupled Reinforcement Learning for Document OCR finds that output entropy is an order of magnitude higher on tables and formulas than on plain text, but uses that entropy only to filter training data offline. BBox-DocVQA measures the gap between answer accuracy and grounding but attaches no uncertainty to either.

Research Program

We define the three signals with the notation used throughout. Let $q$ be the question, $I$ the full page image, $a$ the model's greedy answer, $b$ the evidence box the model emits before answering, and $C(b)$ the image cropped to $b$ with a small margin.

The first signal is answer confidence. We use the greedy negative log-likelihood $A = -\log p(a \mid I, q)$, summed over answer tokens and not length-normalized. We follow Rethinking Uncertainty Estimation in LLMs, which supplies this method: it shows the unnormalized single-sequence score approximates maximum sequence probability and beats sampling-based semantic entropy on text QA.

The second signal is grounding support, the answer's log-probability under teacher forcing (fixing the output tokens and reading off their probability) when the model sees only its claimed evidence:

$$S = \log p(a \mid I, q) - \log p(a \mid C(b), q).$$

A large positive $S$ means the answer depended on content outside the claimed box. That is the measurable trace of right answer, wrong reason.

The third signal is reading confidence, $R$. The model greedily transcribes $C(b)$, and $R$ is the maximum per-word surprisal (how improbable each word was to the model) over that transcription. We add the word-level varentropy, the variance of surprisal, which LogitScope associates with a model torn between competing continuations. We use a word-level maximum rather than a sequence sum because a single illegible word is what ruins a document answer. The choice is also motivated by the finding in Evaluating LLM Uncertainty in Long-Form Generation that sequence-level logit scores rank well at coarse granularity but degrade at the atomic level.

The three signals are combined by a three-feature logistic regression fit on a small held-out calibration split. Every baseline receives the same calibrator. The total cost is roughly three short forward passes, compared with ten or more full samples for semantic entropy.

We deliberately avoid trusting the probabilities of the coordinate tokens themselves. Coordinates written as digits are a known weak point: [Gap 1 in the file] and the spatial-reasoning literature note that turning continuous coordinates into discrete text tokens disrupts geometric precision. Re-scoring with the crop sidesteps that problem. Coordinate-token entropy is kept as an ablation to check whether we are right to avoid it.

The degraded regime is built by applying the Blur-OCR corruption families to BBox-DocVQA pages, using the released code at NikoGuan/Uncertainty_OCR. Those families are Gaussian blur, resolution loss, additive noise, stains and ink bleed, each at four severities. Because each corrupted page has a clean original and a ground-truth evidence box, we can run oracle interventions on every error. Showing the clean page tests whether the error was a reading failure. Showing the ground-truth crop tests whether it was a locating failure. These interventions give labels for attribution, not only for correctness.

Baselines

The naive reference point is answering everything with no abstention, which fixes accuracy at full coverage. Alongside it we report a random abstention ranking, the floor for every selective-prediction curve.

The strongest cheap competitor is answer-only greedy log-likelihood ($A$ alone), run on the same model, prompts and pages. It is the strongest because it is our method minus the two new ingredients. Rethinking Uncertainty Estimation in LLMs reports an average AUROC of 0.721 for it, against 0.690 for semantic entropy, on TriviaQA, SVAMP and NQ-Open. That result is on text-only QA. We found no reported result on Lacuna for any uncertainty method on BBox-DocVQA or degraded DocVQA, so all document numbers will be produced here under identical conditions.

The strongest expensive competitor is semantic entropy over ten sampled answers on the full page. It gets the same calibrator and is run on every example, despite about ten times the decoding compute.

For error-detection power we also run TokUR (TokUR), which perturbs attention weights to estimate epistemic uncertainty. It reports an AUROC of 80.64% on MATH500 with Llama-3.2-1B, against 55.41% for log-likelihood. Its runtime cost confines it to a 1,000-question subset, where it serves as an upper reference rather than a deployable competitor.

For the reading component specifically, the competitor is the tag-trained Qwen2.5-VL-7B of Teaching VLMs to Admit Uncertainty. It reports a word-level uncertainty F1 of 0.685 on Blur-OCR. We compare our $R$ against its tags on Blur-OCR word-level F1, using that paper's own protocol, with the threshold on $R$ chosen on the calibration split.

Three ablations isolate the ingredients: $A+R$ without $S$, $A+S$ without $R$, and $S$ replaced by coordinate-token entropy. Beating the competition means a higher error-detection AUROC and a lower area under the risk–coverage curve than both answer-only log-likelihood and semantic entropy, with non-overlapping bootstrap intervals, on degraded pages at severity 3 and above. The $A+S$ ablation must account for most of that gain on errors labelled as locating failures.

Why not just…?

Why not just use answer log-likelihood, or semantic entropy, which already work for text QA? This is the real threat. If spatial failures already lower answer probability, $S$ adds nothing. There is a specific reason to doubt it: BBox-DocVQA's right-answer-wrong-reason pattern means answers produced from global cues can be fluent and confident, much like the "convincingly wrong" reasoning models in Evaluating LLM Uncertainty in Long-Form Generation. The $A$-only versus $A+S$ comparison, broken down by the oracle locating-error label, settles this directly. If the two are tied, we will report that the decomposition is diagnostic but not predictive.

Why not just restore the image first? Restoration models such as DocPure and DocWaveDiff produce no confidence signal. Where information is truly lost, generative restoration risks the fluent fabrication that Teaching VLMs to Admit Uncertainty documents. Restoration also cannot fix a locating error. We run DocPure as a preprocessing condition and report whether it reduces reading-labelled errors without touching locating-labelled ones, which is the pattern our attribution predicts.

Why not just run an OCR engine and give the text to a strong text model? OCR-Reasoning shows that removing the image and keeping only the OCR text lowers performance. DocVQA shows that flattening a page into one line of text discards layout. A text-only pipeline therefore throws away exactly the spatial information whose failure we want to detect.

Why not just ask the model how confident it is? Evaluating LLM Uncertainty in Long-Form Generation finds that logit scores beat verbalized confidence across its metrics, and zero-shot GPT-4o reaches only 0.039 uncertainty F1 on Blur-OCR. We include verbal confidence as a baseline anyway, because it is the only option for closed models.

Proposed Experiments

The main study uses Qwen2.5-VL-7B and Qwen2.5-VL-72B, the latter reported at 35.20% mean IoU on BBox-DocVQA. It covers the single-page single-box split of BBox-DocVQA at five severities: clean plus four corruption levels. The single-page multi-box split uses the union of predicted boxes as the crop. We report error-detection AUROC, area under the risk–coverage curve, and expected calibration error.

The first decisive check is attribution. For each error we record whether the clean page fixes it and whether the ground-truth crop fixes it. We then test whether $R$ is higher on errors fixed by the clean page and $S$ is higher on errors fixed by the crop, measured as an AUROC for each signal against its matching oracle label. Our prediction is a crossover: as severity rises, the share of reading-labelled errors grows, and the weight the calibrator places on $R$ grows with it.

The second decisive check is transfer to natural noise without refitting. We use FUNSD's real scanned forms, where OCR reaches 94.4% similarity but key–value linking collapses to an F1 of 0.04 (FUNSD), with linking recast as questions about which value belongs to a given key. We also use the spatial-analysis subset of OCR-Reasoning, where spatial comprehension accounts for 27.2% of errors. Here we test whether $S$ still separates errors when no clean original exists.

The third check is a failure mode of the method itself. When the claimed box is too tight and drops needed context, $S$ will flag correct answers. We measure the false-flag rate on correct answers as a function of the margin added to the crop, and report the margin at which the gain over answer-only log-likelihood disappears.