Systems that pull fields such as dates, totals, and vendor names out of invoices and forms work well on document templates they were trained on. They lose 13 to 17 F1 points when the template is new (A Benchmark for Structured Extractions from Complex Documents). In deployment nobody labels the new template, so nobody notices which extracted fields are wrong. This note proposes a label-free check for each extracted field. We take the document and rewrite the layout description given to a language model in ways that keep the page's meaning intact: coarser or finer coordinates, a different reading order, a different coordinate notation, or a small shift of the whole page. We then check whether the extracted value survives. A model that has learned where a field usually sits will flip under these rewrites. A model that reads the label-to-value structure of the page will not.
The proposal is a step past a known result. Instability under rewordings of the instruction predicts errors in plain-text classification, and On the Relation between Sensitivity and Accuracy in In-Context Learning turned this into an abstention rule called SENSEL. On a new template, though, the failure lives in the layout, not in the instruction. Our hypothesis is that rewording the layout is the right probe for layout failures, and that rewording the instruction is the wrong one. If this holds, it yields a practical tool: deciding which fields to send for human review on an unfamiliar form. It also yields a measurement: how much of a model's behaviour on seen templates is coordinate memorization.
Core Claim
We study LLMs that read layout as text, in the style of LMDX. That system appends normalized coordinate tokens to each OCR span (for example Apple Store 38|05) and asks the model to emit JSON fields with coordinate anchors. For such models, we claim that disagreement across meaning-preserving layout rewrites predicts which fields are wrong on unseen templates. We claim it does so better than three alternatives: disagreement across instruction paraphrases, disagreement across random samples, and token probability.
Crucially, the claim must hold within a single unseen document, not only across documents. Otherwise it collapses into "new templates are harder," which a cheap layout-novelty detector could report on its own. A second, descriptive claim follows. For a model fine-tuned on some templates, the gap between layout-rewrite sensitivity on seen and unseen templates should track that model's accuracy drop on unseen templates. This would give a per-model estimate of layout memorization that requires no unseen-template labels.
What Motivates and Constrains This
Two benchmarks establish the gap. VRDU's Unseen Template Learning split shows models behaving "more as memorizers than understanders" (A Benchmark for Structured Extractions from Complex Documents). DocILE reports a clear drop on zero-shot layouts: its best model reaches 0.698 against a 0.946 upper bound for extraction approaches that tag text spans (DocILE).
Two lines of work explain why instability should mark errors. ProSA links prompt sensitivity to low decoding confidence on text benchmarks. SENSEL (On the Relation between Sensitivity and Accuracy in In-Context Learning) shows a correlation of up to −0.40 between sensitivity and accuracy, and gains 4.1 points on its selective-prediction metric over raw softmax probability on GPT-J-6B. Neither paper touches layout. Both perturb only the wording of the instructions or the order of the in-context examples, never the serialization of the input.
Four document-extraction papers supply methods but leave this question open. QueryForm (QueryForm) explicitly names prompt sensitivity as unresolved future work. P²Net (P²Net) links values to visual keys with pointers, but never perturbs its queries. LMDX uses majority voting purely to correct errors and does not examine what disagreement reveals. InstructDoc reports lower variance across instruction phrasings for its model. That is a coarse robustness check, not a study of individual fields on unseen templates.
One result in this group cuts against us. ProSA finds that a single in-context example sharply reduces sensitivity. If one example also removes layout-rewrite sensitivity without fixing errors, our signal would be useless in the few-shot regime. We test this directly.
Research Program
We first build a set of layout rewrites that keep the document's meaning, and check that they really do. Coordinate quantization varies the grid on which positions are rounded, from coarse to fine. Reading order switches between top-to-bottom left-to-right, column-first, and the OCR engine's native order. Coordinate notation switches between x|y, explicit bounding boxes, and row/column bins. Global shifts translate or scale the whole page by a small amount. To check validity, we confirm that a fine-tuned model's accuracy on seen templates is essentially unchanged under each rewrite. Any rewrite that drops seen-template accuracy by more than one F1 point is discarded. As a negative control, we add a rewrite that shuffles coordinates across tokens. It must destroy accuracy; if it does not, the model is not using layout at all and the study is moot for that model.
For each extracted field we then compute two quantities over K forward passes. The first is the layout-rewrite agreement rate: the fraction of rewrites that return the same normalized value. The second is the instruction-paraphrase agreement rate. Its paraphrases follow the four ProSA template families: simple input, emotional support, role player, and output requirement. In both cases a field is treated as missing when it is absent from the output, so the rule handles the absent-entity case that QueryForm flags.
We run this in two regimes. The first is pure zero-shot, using open instruction-tuned models. Llama-3.3-70B and Qwen-2.5-32B, the models used in ZeroSemble (DocIE@XLLM25: ZeroSemble), give us access to token probabilities. The second is a smaller open model fine-tuned LMDX-style on seen templates, so that seen and unseen splits exist. The zero-shot constraint governs the detector, not the extractor. No labels or exemplars from the target template are ever used.
Baselines
The naive reference point is to accept every field from a single greedy pass. Its F1 on each split anchors every coverage curve.
The cheapest confidence baseline is the mean token probability of the generated value. ProSA ties sensitivity to exactly this kind of decoding confidence. If token probability already captures the signal, multi-pass probing is unnecessary.
The strongest existing selective-prediction method is SENSEL (On the Relation between Sensitivity and Accuracy in In-Context Learning), applied at the field level with instruction paraphrases. Its reported result is +4.1 points on the area under the F1-versus-coverage curve over softmax probability, averaged across text classification datasets on GPT-J-6B. No reported result was found on Lacuna for SENSEL or any sensitivity-based abstention method on document extraction.
The second strong competitor is sampling agreement: K samples at nonzero temperature, using the same prompt and the same layout serialization, scored as in LMDX majority voting.
The last competitor is a layout-novelty score. It is the bounding-box image distance from the test document to its nearest training template, computed as in SAIL (SAIL). It requires one pass and no rewrites.
Fairness and the definition of a win are fixed in advance:
- Matched compute: all multi-pass methods use the same K (we report K = 4, 8, 16), the same models, and the same OCR output.
- Tuning: paraphrase and rewrite sets are fixed on a validation split before any test scoring.
- Ablations: layout rewrites alone, instruction paraphrases alone, and an equal mix at matched K. This isolates whether the layout channel itself is the key ingredient.
- What counts as beating SENSEL: at least a 3-point gain in area under the F1-coverage curve on unseen-template fields, the same order as SENSEL's own reported gain over softmax probability. The gain must hold with documents fixed, as described under the decisive checks.
Why not just…?
Why not just use sampling agreement, which LMDX already does? This is the most serious threat. Sampled outputs from a flat next-token distribution will disagree on exactly the fields where the model is unsure. If sampling agreement matches layout-rewrite agreement at equal K, we will report that, and the contribution shrinks to a negative result about what perturbation adds. Our expectation that it will not match is specific. A model confidently reading a memorized position can be sharp, and therefore consistent under sampling, while still wrong. A layout rewrite moves the cue being memorized. The matched-K comparison restricted to confidently wrong fields (high token probability, incorrect) settles this.
Why not just flag unfamiliar templates with a layout-distance detector? Such a detector ranks documents, not fields, and in many cases document-level triage may suffice. The proposal survives only if layout-rewrite agreement separates correct from incorrect fields within the same unseen document. It must also add predictive power when combined with the novelty score, beyond the novelty score alone. We test both explicitly. Failing either would mean the cheaper detector should be used instead.
Why not just add one in-context example, since ProSA shows this stabilizes models? In the target setting there is no labeled example of the new template. A labeled example from some other template may stabilize outputs in the wrong direction. We run the 1-shot condition with an example drawn from a different template. If it removes both the errors and the layout sensitivity, few-shot anchoring is the better fix and we say so. If it removes the sensitivity but not the errors, the anchor masks failures, and the detector should run before the anchor is added.
Why not just use a bigger multimodal model? The file's evidence argues against scale as a fix. GPT-4V with OCR text scores 66.22 F1 zero-shot on CLEX, against 74.04 for the far smaller P²Net (P²Net). ProSA finds GPT-4o more prompt-sensitive than Llama 3-70B-Instruct. Larger models still need a failure detector.
Proposed Experiments
The main testbed is VRDU (A Benchmark for Structured Extractions from Complex Documents). We compare its Single Template and Unseen Template splits on Registration Forms, where the reported unseen-template drop is 13 to 17 points, and on Ad-buy Forms, where micro-F1 sits at 40 to 46% and line-item tables stress hierarchical structure. We add DocILE's zero-shot-layout test portion (DocILE) and the zero-shot CLEX split (P²Net) for cross-lingual breadth.
The metrics are:
- area under the F1-versus-coverage curve;
- selective F1 at 80% coverage;
- the correlation between each field's agreement rate and its correctness;
- for the descriptive claim, the correlation across fine-tuning checkpoints and model sizes between the seen-versus-unseen gap in layout sensitivity and the seen-versus-unseen F1 gap.
The decisive checks are three. First, layout-rewrite agreement must beat SENSEL, sampling agreement, and token probability at matched K by the stated margin on unseen-template fields. Second, the gain must persist in a within-document analysis that pairs correct and incorrect fields from the same unseen document, which rules out the novelty explanation. Third, the shuffled-coordinate control must collapse accuracy, while the validity-filtered rewrites leave seen-template accuracy intact.
A secondary analysis addresses the open question of whether instruction sensitivity itself rises on unseen layouts. We report ProSA-style sensitivity separately for seen and unseen templates. If it does not rise while layout sensitivity does, that cleanly separates two kinds of brittleness that the literature currently treats as one.
References
- VRDU: A Benchmark for Visually-rich Document Understanding
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- LMDX: Language Model-based Document Information Extraction and Localization
- DocILE Benchmark for Document Information Localization and Extraction
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- QueryForm: A Simple Zero-shot Form Entity Query Framework
- P²Net: Parallel Pointer-based Network for Key Information Extraction with Complex Layouts
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions
- DocIE@XLLM25: ZeroSemble - Robust and Efficient Zero-Shot Document Information Extraction with Heterogeneous Large Language Model Ensembles
- SAIL: Sample-Centric In-Context Learning for Document Information Extraction