Language models that read business documents can now point to where an answer sits on the page. One recipe, LMDX, writes each OCR word followed by its quantized page position (for example Apple Store 38|05) and lets a text model emit those position tokens alongside the extracted value. These models are trained with ordinary next-token cross-entropy, a loss that scores a guess only by whether it hit the exact correct token. Under that loss, predicting position bin 39 when the truth is 38 costs as much as predicting 99. Evaluation has the same blind spot: an invoice field whose box misses a binary overlap threshold by a few pixels counts as a total failure. This proposal asks whether that shared blindness to distance explains part of the well-documented collapse of document models on layouts they have never seen. It also asks whether the fix belongs in domain pre-training, where millions of unlabeled documents are available, rather than in the few hundred labeled examples used for fine-tuning.
The plan has two steps. First, we re-score existing domain-adapted document models on DocILE's unseen-layout split with distance-aware metrics. These separate "found the right region but drew a sloppy box" from "looked in the wrong place." Second, we continue pre-training a text LLM on DocILE's roughly one million unlabeled business documents with a self-supervised task: hide the position tokens of some text lines and predict them from the surrounding words and positions. The prediction is scored with a loss that gives partial credit to nearby bins. If the unseen-layout drop is mostly sloppy boxes, this pre-training should shrink it. If the drop is mostly wrong regions, the measurement study will say so, and that is itself a useful negative result.
Core Claim
Our claim is that distance-aware supervision of coordinate tokens matters most during domain pre-training, not during fine-tuning. Pre-training is where the model learns the geometry of a document family: where totals sit relative to line items, how columns align. Fine-tuning labels are too few to teach that geometry. Concretely, we predict that a model pre-trained with distance-aware masked-coordinate prediction will beat an identically pre-trained model trained with plain cross-entropy on DocILE's unseen layouts. We predict the gain will appear mainly in the localization-error component of the score, not in false negatives. We also predict that applying the same soft loss only at fine-tuning time recovers little of that gain.
The loss itself is deliberately simple. For a hidden horizontal (or vertical) coordinate with true bin $c$ among $K$ bins, we replace the one-hot target with
$$q(k) = \frac{\exp(-|k-c|/\tau)}{\sum_{j=1}^{K}\exp(-|j-c|/\tau)},$$
where $k$ indexes candidate bins and $\tau$ is a width in bins. We then minimize the cross-entropy between $q$ and the model's predicted distribution over coordinate tokens. As $\tau \to 0$ this reduces to LMDX's loss. A larger $\tau$ gives graded credit to near misses, and every bin receives a non-zero gradient. That avoids the zero-gradient plateau that Directly Optimizing IoU for Bounding Box Localization documents for raw overlap losses on non-overlapping boxes. Horizontal and vertical positions get separate one-dimensional targets, matching how LMDX encodes position.
Relation to Prior Work
Three papers supply the setting and method. LMDX supplies the coordinate-as-text format and the generative extraction interface. It trains coordinates with plain cross-entropy and only on labeled extraction tasks; we reuse its format but add a self-supervised pre-training stage and a distance-aware target. LayoutLMv3 pre-trains on 11 million scanned documents. Its only spatial objective, word-patch alignment, is a binary "is this patch masked" classifier, so nothing in its pre-training rewards being geometrically close. DocILE supplies the data: 6,680 annotated documents, 100k synthetic documents, and nearly a million unlabeled ones. It reports a sharp drop on unseen layouts, but its metrics count a prediction as a hit only above an overlap threshold, and it leaves the unlabeled set unused.
Two lines of work motivate the measurement half. SMuDGE shows that character-similarity scores reward answers that do not physically exist on the page, and it scores grounding with a normalized Manhattan distance between boxes. It is evaluation-only and covers extractive question answering, not domain pre-training. LRP splits detection error into box tightness, false positives and false negatives. That split is exactly what we need to tell sloppy boxes from wrong regions. The distance-curve area used in geoparsing evaluation gives a threshold-free summary that we adapt to page coordinates.
One paper constrains the design. Correlation Loss finds that when a model cannot produce tight boxes, forcing its confidence to track localization quality can hurt. We therefore do not touch confidence calibration in training. We also treat the box-tightness component of LRP as a precondition: if pre-training does not tighten boxes, downstream gains are not expected.
Domain pre-training work frames the bet but does not settle it. MonoCoder and BERTHop show that in-domain pre-training beats larger general models. A Comparative Study of PEFT Methods for Python Code Generation finds that pre-training predicts downstream performance better than the choice of fine-tuning method. These are text-only results, and whether they transfer to spatial grounding is precisely what is untested.
Research Program
The first stage is diagnostic. We fine-tune an LMDX-style model and LayoutLMv3 (with DocILE's synthetic-document pre-training) on DocILE's training set. We then score both on the seen-layout and unseen-layout test documents with four measures: the official KILE and line-item F1, the three LRP components, the normalized Manhattan distance used in SMuDGE, and the area under the cumulative distance-error curve. The quantity of interest is the share of the unseen-layout F1 drop that moves into the box-tightness component versus the false-negative component. This stage decides whether the training intervention has room to help.
The second stage is the intervention. We take Llama-3.1-8B, the open base used by FeynTune for domain specialization. We OCR DocILE's unlabeled documents into the LMDX serialization and continue pre-training with a mixture of two parts. The first is next-token prediction on the serialized page. The second is masked-coordinate prediction, in which the position tokens of a random 15% of lines are replaced by a mask symbol and must be predicted. We run three pre-training arms on the same token budget and schedule. The first uses plain cross-entropy on hidden coordinates. The second uses the distance-aware target with $\tau$ tuned on a held-out layout cluster. The third uses no masked-coordinate task at all, only next-token prediction on the serialized pages. Each arm is then fine-tuned identically on DocILE's labeled data, in two versions: once with plain cross-entropy and once with the soft target at fine-tuning time.
Baselines
The naive reference is the off-the-shelf base LLM fine-tuned in the LMDX format with no domain pre-training. This shows what the domain stage buys at all, and it is run with the same fine-tuning data, steps and learning-rate sweep as every other arm.
The strongest published competitors on DocILE are LayoutLMv3 boosted with 100k synthetic documents and the best DocILE baseline. DocILE reports LayoutLMv3 at KILE F1 0.655 and line-item F1 0.691, with the best baseline reaching KILE 0.698. We rerun LayoutLMv3 with the released recipe so it is scored under the new continuous metrics on identical splits. Beating it means exceeding 0.698 KILE on the official metric, or matching it while showing a lower box-tightness error on unseen layouts.
For the LMDX recipe itself, the closest method, LMDX reports line-item F1 of 72.09 on VRDU Ad-buy with 200 training documents, against 25.46 for LayoutLMv2. Its numbers come from PaLM 2-S, which we cannot run. We therefore reimplement the recipe on our base model and compare against our own reimplementation, quoting the published figure only as context.
The ablation that isolates the key ingredient is the second against the first pre-training arm, distance-aware against plain cross-entropy, with everything else held fixed. The arm without masked-coordinate prediction separates the effect of the self-supervised coordinate task from the effect of the loss shape. Any claimed improvement must exceed seed-to-seed variance across three seeds.
Why not just…?
Why not just apply the soft coordinate target during fine-tuning and skip domain pre-training? This is the most serious threat. It is a one-line change, and if it captures most of the gain our pre-training story is unnecessary. The design measures it directly: the plain-pre-training arm fine-tuned with the soft target is the comparison. The only indirect evidence that pre-training matters more is text-only, from MonoCoder and the PEFT study, so we do not presume the answer. If fine-tuning-time smoothing closes most of the gap on unseen layouts, we will report that the cheap fix suffices and the pre-training claim fails.
Why not just change the metric? If the unseen-layout drop disappears under distance-aware scoring, that is half of our finding, and stage one tests it directly. It would not by itself make models more useful, because a downstream system still needs the right box. And SMuDGE shows that metric choice can also hide failures, not only exaggerate them. We report both readings.
Why not just use a large frozen multimodal model with an external text detector? DLaVA does this with Pixtral-12B and a separate text detector, reaching 85.91 answer similarity (ANLS) on DocVQA but only 46.22 mAP over overlap thresholds 0.50 to 0.95. That gap between finding the answer text and placing its box is the failure we target. DLaVA cannot update its backbone from spatial errors. We include it as a training-free reference on DocVQA.
Why not just add a continuous box-regression head? This is feasible in encoder models like LayoutLMv3. In a generative text model it breaks the single-output interface that makes LMDX sample-efficient. The soft-label target keeps that interface while giving the graded credit a regression loss would give.
Proposed Experiments
The decisive comparison is on DocILE's unseen-layout documents. The distance-aware pre-training arm must beat the cross-entropy pre-training arm on KILE and line-item F1. The improvement must show up as a smaller box-tightness component in LRP and a larger area under the distance-error curve, not merely as fewer false negatives. It must also be clearly larger than the improvement from soft targets applied only at fine-tuning.
Two secondary checks test generality. The first is transfer to VRDU Ad-buy and Registration with 10 and 200 training documents, to test whether the effect carries to an unseen document family. The second is DocVQA scored with SMuDGE's grounding term at $\alpha = 0.25$, where SMuDGE reports no per-model numbers, against DLaVA's reported localization score, to test whether better coordinate geometry also reduces answers with no physical grounding.
The outcome could falsify the premise. If stage one shows the unseen-layout drop is dominated by false negatives, distance-aware coordinates cannot be the main lever. In that case we will report the decomposition as the contribution and drop the intervention claim.
References
- Directly Optimizing IoU for Bounding Box Localization
- LMDX: Language Model-based Document Information Extraction and Localization
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
- DocILE Benchmark for Document Information Localization and Extraction
- Where is this coming from? Making groundedness count in the evaluation of Document VQA models
- Localization Recall Precision (LRP): A New Performance Metric for Object Detection
- A pragmatic guide to geoparsing evaluation
- Correlation Loss: Enforcing Correlation between Classification and Localization
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
- A Comparative Study of PEFT Methods for Python Code Generation
- FeynTune: large language models for high-energy theory
- DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness