Multilingual document models read a scanned form or invoice by pairing each text token with a visual feature cut out of the page image. That visual feature usually comes from a coarse feature map, often one cell per 16 or 32 pixels, that is pooled or bilinearly stretched to the token's bounding box. On dense pages this breaks down. A Chinese line, a tightly packed table, or a column of short codes puts several tokens inside one coarse cell, so those tokens receive nearly identical, blended visual features. We propose to replace that content-blind step with a content-aware upsampler, meaning an operator that computes its interpolation weights from the features themselves rather than from fixed geometry. Its guidance signal would be built from both the high-resolution image and the known token boxes. Each fine-resolution pixel is then assigned to the coarse feature that matches the token it actually belongs to, instead of an average of its neighbours.
The bet is that this matters most where existing multilingual document models are weakest. Those places are dense scripts, table-heavy domain documents, and cross-lingual transfer, where visual structure such as alignment, ruling lines, and box geometry carries over between languages but token text does not. If the bet is right, the gains should grow with how many tokens collide in one coarse cell. They should also narrow the gap between zero-shot cross-lingual transfer and joint multilingual training. Both predictions can be checked on the XFUND benchmark, the DocILE benchmark, and the MORE benchmark.
Core Claim
Content-aware upsamplers come from dense computer vision. CARAFE predicts a small reassembly kernel at every output location and reports +1.8 mIoU for UperNet on ADE20K. SAPA and its extension On Point Affiliation in Feature Upsampling take a different route. They treat upsampling as "point affiliation": each high-resolution point picks, by similarity to a high-resolution guidance feature, which nearby low-resolution feature it belongs to. SAPA-D reports +2.57 mIoU and +3.58 boundary IoU on ADE20K. These papers supply our method. They also bring a constraint. SAPA needs a high-resolution guidance map, and the follow-up work notes it helps boundary sharpening ("detail delineation") more than filling region interiors ("semantic mending").
Document models offer a guidance signal that natural images lack: the OCR boxes and the identity of the token inside each box. Our claim is that point affiliation guided by a rasterized token-and-layout map gives each token a visual feature that is not contaminated by its neighbours. The image-only guidance used by SAPA cannot supply this. The result should improve token-level entity and relation extraction in proportion to token density, and it should do so more for scripts and documents where density is high. Concretely, let $e_i$ be a cheap high-resolution image feature at fine location $i$ (from the backbone's stride-4 stem), and let $t_{b(i)}$ be the text embedding of the token whose box $b(i)$ contains $i$, or a learned "background" vector. Let $d_j$ be the coarse backbone feature at candidate location $j$ in a $5\times5$ window $N(i)$, and let $P$ and $Q$ be learned projections. The upsampled feature is
$$ \hat f_i = \sum_{j \in N(i)} \frac{\exp\big(\langle P[e_i; t_{b(i)}],\, Q d_j\rangle\big)}{\sum_{k\in N(i)} \exp\big(\langle P[e_i; t_{b(i)}],\, Q d_k\rangle\big)}\, d_j , $$
and each token's visual feature is the mean of $\hat f_i$ over its box. The only new ingredient compared with SAPA is the token term $t_{b(i)}$ in the guidance. Two adjacent tokens with different text, and therefore different guidance vectors, pull from different coarse candidates even when they sit in the same coarse cell.
Research Program
The host model is LayoutXLM. LayoutXLM is a multimodal Transformer pre-trained on 30M pages in 53 languages. It extends LayoutLMv2 with character-level boxes for unsegmented scripts and defines XFUND, a 7-language form-understanding benchmark. Its visual tokens come from a CNN feature map pooled to a fixed grid. Its text tokens get their visual signal only through attention to that grid. We modify the visual branch so that a per-token visual feature, computed with the upsampler above, is added to each token embedding. We do this for three variants: bilinear upsampling, image-guided SAPA, and layout-guided SAPA.
Each direction plays a different role. Content-aware upsampling is the mechanism. Domain-specific pre-training is the regime. We continue pre-training the modified model on unlabeled domain documents, using DocILE's roughly 1M unlabeled business documents plus a multilingual slice of the PIN-14M interleaved document set. We do this rather than inserting the operator only at fine-tuning time, because a new operator inside a pre-trained model may simply be ignored if it never sees pre-training. mDAPT supports this choice: in that work, continued multilingual domain pre-training of a single model beat general mBERT and matched monolingual specialists. Multilingual joint training is the evaluation lens. LayoutXLM reports a gap between zero-shot transfer from English FUNSD (0.6115 average SER F1) and joint training on all eight languages (0.8429). Visual structure is shared across scripts, so cleaner per-token visual features should shrink this gap.
The analysis tool is a per-token collision count: the number of other tokens whose boxes overlap the same coarse feature cell. We compute it for every token in every test set. It lets us test the mechanism directly rather than infer it from average F1. Gains should rise with collision count and should be near zero for isolated tokens.
Baselines
The naive reference is unmodified LayoutXLM-base, fine-tuned on XFUND with the published recipe. The file reports only LARGE numbers: SER 0.8282 and RE 0.7206 average F1 for language-specific fine-tuning, and 0.8429 SER for joint training, all from LayoutXLM. We found no reported LayoutXLM-base XFUND number on Lacuna, so we will reproduce base ourselves and also run the full comparison at LARGE scale for the headline row. A text-and-layout-only lower reference is InfoXLM_LARGE, which the LayoutXLM paper reports at 0.7471 SER and 0.6002 RE.
The strongest competitor is not an upsampler. It is the same LayoutXLM with the visual backbone run at higher input resolution, with per-token features taken by RoIAlign from a stride-8 feature map using bilinear interpolation. This attacks the collision problem by brute force and is the obvious thing a document-AI engineer would try. We will match its training FLOPs to our model by trading resolution against steps, and both get the same continued pre-training data, steps, and hyperparameter search budget of eight learning-rate and warmup configurations per method on the validation split.
The ablation that isolates our key ingredient is image-guided SAPA, which uses the same operator without the token term $t_{b(i)}$. Off-the-shelf CARAFE is included because the point-affiliation papers report it is better at interior mending, which may matter for multi-line table cells. Beating the field means a statistically reliable gain, over three seeds, against the resolution-matched bilinear model on XFUND RE averaged over languages. On DocILE, the bar is to exceed LayoutLMv3's 0.655 KILE and 0.691 LIR F1 and the best reported baseline of 0.698 KILE from DocILE. These are English business documents, so DocILE tests the domain regime rather than multilinguality.
Why not just…?
Why not just raise the image resolution and keep bilinear pooling? This is the real threat, and the proposal is designed to settle it rather than argue around it. Higher resolution reduces collisions for everyone but costs compute quadratically. The layout-guided operator reuses an existing stride-4 stem and adds a $5\times5$ local softmax; CARAFE's own accounting puts that class of operator at about 199k FLOPs versus 1,180k for deconvolution for a 2× upsample. The test is the FLOP-matched comparison above, broken down by collision count. If high-resolution bilinear matches us in the high-collision bins, the content-aware component is unnecessary and we will report that.
Why not just use SAPA or CARAFE as published, with image features as guidance? On a page, neighbouring glyphs of different tokens often look alike: same font, same ink, same stroke density. Image similarity alone may therefore not separate them. That failure is the motivation for the token term, but it is a hypothesis, and the image-guided SAPA ablation measures it directly. If image guidance matches layout guidance, the contribution reduces to "SAPA helps document models," which is still new per the research file but is narrower.
Why not just drop vision and rely on text plus box coordinates? For dense tokens, the box coordinates already encode position exactly, so one could argue visual features add little. The LayoutXLM paper's InfoXLM comparison shows text-only models lose about 8 SER and 12 RE points on XFUND. That establishes vision helps overall, but not that it helps for collided tokens specifically. Our collision-binned analysis includes a variant with the visual branch zeroed, to show where visual features matter.
Why not just translate to English and run an English document model? LayoutXLM argues that translation changes token counts and breaks box alignment. En-ViMedNER finds, in a domain setting, that translation quality does not track entity-extraction accuracy and that target-language supervision beats English transfer by over 16 F1. We do not run translate-test, because it cannot produce token-aligned layout output for XFUND's relation task.
Proposed Experiments
The primary experiment is XFUND SER and RE in three regimes: language-specific fine-tuning, zero-shot transfer from FUNSD, and joint eight-language training. All five model variants are run at base scale with three seeds, and the naive, resolution-matched, and layout-guided variants are repeated at LARGE scale. We report per-language scores because the file gives only averages, and we expect Chinese and Japanese to gain most. The domain experiment uses DocILE KILE and LIR after continued pre-training on its unlabeled split. A layout-segmentation check on PubLayNet, where LayoutLMv3 reports 95.1 mAP in LayoutLMv3, tests whether the operator also helps a native dense-prediction head. Finally, the table and raw-page subsets of MORE serve as a held-out multilingual stress test for table cell detection. MORE reports table parsing as the collapse point, with dots.ocr at 39.81%. We use it only as a relative comparison between our variants, since our model is not a full parser.
Three decisive checks determine whether the claim stands. First, gains must increase monotonically with collision count and be indistinguishable from zero for isolated tokens. If gains are flat across bins, the mechanism we describe is not what is helping. Second, layout-guided must beat image-guided SAPA in the high-collision bins. Otherwise the token term contributes nothing. Third, the reduction in the zero-shot-versus-joint gap on XFUND should be larger than the in-language gain. This would indicate that cleaner visual grounding transfers across scripts. A secondary check fine-tunes the new operator without continued pre-training. If it matches the pre-trained version, the domain pre-training stage is unnecessary for this operator, which is useful to know given the roughly ten GPU-days such stages cost in mDAPT.
References
- CARAFE: Content-Aware ReAssembly of FEatures
- SAPA: Similarity-Aware Point Affiliation for Feature Upsampling
- On Point Affiliation in Feature Upsampling
- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
- mDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
- DocILE Benchmark for Document Information Localization and Extraction
- En-ViMedNER: An English-Vietnamese Parallel Biomedical Corpus with UMLS Semantic Type Annotations
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
- MORE: A Multilingual Document Parsing Benchmark and Evaluation