Benchmarking Multilingual Document Understanding in Vision-Language Models

Matteo Gabburo, Daniela Baiamonte, Stefano Simonazzi, Silvia Gardini, Sean Cesare Lazzeri, Andrea ZugariniarXiv 2026 · arXiv

TL;DR

The authors introduce MDU, a comprehensive multilingual benchmark for Document Understanding (DU) that spans 13 tasks across five languages. By evaluating 45 state-of-the-art Vision-Language Models (VLMs), they reveal that current models struggle with layout reasoning and multi-page documents, and that increasing model scale does not close the performance gap between languages.

What Problem Does This Paper Address?

Document Understanding has evolved from fragile pipelines of OCR and layout analysis to end-to-end Vision-Language Models (VLMs). While these models are powerful, our ability to evaluate them is lagging. Most existing benchmarks are English-centric and focus on narrow tasks like visual question answering (VQA).

This creates a "blind spot" in research: we don't know how well models generalize to other languages, diverse document styles (like legal forms vs. scientific posters), or complex multi-page layouts. Furthermore, it’s unclear if a model’s ability to transcribe text actually correlates with its ability to understand a document's semantic structure.

Methods

The researchers developed a synthetic but human-validated pipeline to create the MDU benchmark.

  1. Data Sourcing: They sampled 2.5 million documents from the FinePDFs corpus, filtering for permissive licenses and high-quality renders across English, French, German, Italian, and Spanish.
  2. Synthetic Generation: They used a high-performing VLM (Qwen3.5-122B-A10B) to generate instruction-response pairs grounded in the document images. To ensure variety, they used "persona-based" prompting to vary the style and tone of instructions.
  3. Human Audit: Native linguists audited 3,000 examples to ensure the synthetic references were sound. They found a 90%+ acceptance rate, making the benchmark a reliable ground truth.
  4. Evaluation Protocol: They benchmarked 45 open-weight models (ranging from 1.7B to 235B parameters) and one commercial API (Gemini 3.8 Flash).
  5. Novel Metrics: For layout extraction, they moved away from standard geometric Intersection-over-Union (IoU) scores, which unfairly penalize models for tiny coordinate offsets on thin lines of text. Instead, they scored models based on the actual words covered by the predicted bounding boxes.
MDU at a glance: 13 tasks across 5 languages and 77 domains.

Datasets

The primary contribution is the MDU Dataset:

  • Size: 23,146 examples.
  • Languages: German (de), English (en), Spanish (es), French (fr), and Italian (it).
  • Tasks: 13 tasks including Text/Markdown Extraction, Layout Extraction, Named Entity Recognition (NER), Classification, QA, Summarization, and Translation.
  • Complexity: Includes both single-page and multi-page (2-5 pages) documents.
A breakdown of the 13 tasks, metrics used, and language distribution.

Key Insights and Ideas

  • Scale is not a Silver Bullet: Surprisingly, the best-performing models were 27B parameter versions (Qwen3.6 and Qwen3.5), which outperformed massive models like the 235B Qwen3-VL. Model family architecture seems to be a better predictor of success than raw parameter count.
  • The Persistent Language Gap: English remains the "easiest" language for models, while German is consistently the hardest. Critically, this gap does not shrink as models get larger; scale lifts all boats but keeps the distance between them the same.
  • Skill Independence: A model’s ability to transcribe text (OCR) is largely uncorrelated with its ability to perform semantic extraction (like NER). This suggests that "document understanding" is not a monolithic skill but a collection of separable capabilities.
  • The Multi-Page Penalty: Processing multiple pages is significantly harder for VLMs. Performance drops sharply on tasks like text extraction and translation when the model has to maintain coherence across page boundaries.
Figure 2: Single-page versus multi-page performance, averaged over the 45 evaluated models. Bars are means, whiskers one standard deviation across models. Tasks are ordered by the size of the drop. The nine tasks with paired variants are shown; Layout, NE and Pas- sage Extraction and Conditional Summarization exist in single-page form only. The two classification tasks are the only ones where the multi-page variant is easier.

Key Results

  • Benchmark Saturation: The suite is far from solved. The highest average score was only 69.0 (Qwen3.6 27B). Only eight out of 45 models managed to score above 65.
  • Hardest Tasks: Layout Extraction and Entity Extraction proved to be the most difficult, with average scores across all models hovering around 34-35%.
  • Commercial vs. Open-Weight: Gemini 3.8 Flash performed well (65.8 average, 7th overall) but was notably weaker at raw transcription tasks compared to specialized open-weight models.
Detailed per-task scores for top models across different parameter scales.

Limitations and Future Work

  • Linguistic Scope: The study is limited to five European languages using the Latin script. The authors acknowledge that these findings might not generalize to low-resource languages or non-Latin scripts (e.g., Arabic, Chinese).
  • Synthetic Bias: Because a Qwen model was used to generate the benchmark, there is a risk of "self-preference" where Qwen-family models score higher due to stylistic similarities. The authors quantified this and found it accounts for about +7.8 points, which readers should keep in mind when viewing the leaderboard.
  • Future Directions: The authors suggest that future work should focus on improving layout-grounded spatial reasoning and multi-page coherence, which remain the primary bottlenecks for current VLMs.

Summary updated

Abstract

Recent Vision-Language Models (VLMs) perform strongly on multimodal tasks, Document Understanding (DU) among them, but the benchmarks that measure them mostly cover English documents and narrow problems. How well current VLMs generalize across languages, tasks, styles and layouts therefore remains unclear. We introduce MDU, a multilingual DU benchmark of 23146 examples spanning 13 tasks and five languages: English, French, German, Italian, and Spanish. The tasks isolate complementary capabilities, from surface-level text and Markdown extraction through summarization, classification and named-entity recognition to layout-sensitive passage and layout extraction. We evaluate 45 state-of-the-art open-weight VLMs from 1.7B to 235B parameters, plus one commercial system under the same protocol. The benchmark is far from saturated: the best average is 69.0, only eight of the 45 evaluated systems reach 65, and the two best results come from 27B models rather than from the largest systems we evaluate. Performance is uneven in ways that scale does not fix. The gap between the strongest and weakest language does not close as models grow, and it survives controlling for the domain composition of each language. Capabilities also do not improve together: once we partial out overall model quality, per-task scores are largely uncorrelated, and transcription ability in particular trades off against semantic extraction.
BibTeX
@misc{gabburo2026benchmarking,
  author = {Matteo Gabburo and Daniela Baiamonte and Stefano Simonazzi and Silvia Gardini and Sean Cesare Lazzeri and Andrea Zugarini},
  title = {{Benchmarking Multilingual Document Understanding in Vision-Language Models}},
  year = {2026},
  eprint = {2611.14758},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14758},
}
Figure