TL;DR
The authors introduce MDU, a comprehensive multilingual benchmark for Document Understanding (DU) that spans 13 tasks across five languages. By evaluating 45 state-of-the-art Vision-Language Models (VLMs), they reveal that current models struggle with layout reasoning and multi-page documents, and that increasing model scale does not close the performance gap between languages.
What Problem Does This Paper Address?
Document Understanding has evolved from fragile pipelines of OCR and layout analysis to end-to-end Vision-Language Models (VLMs). While these models are powerful, our ability to evaluate them is lagging. Most existing benchmarks are English-centric and focus on narrow tasks like visual question answering (VQA).
This creates a "blind spot" in research: we don't know how well models generalize to other languages, diverse document styles (like legal forms vs. scientific posters), or complex multi-page layouts. Furthermore, it’s unclear if a model’s ability to transcribe text actually correlates with its ability to understand a document's semantic structure.
Methods
The researchers developed a synthetic but human-validated pipeline to create the MDU benchmark.
- Data Sourcing: They sampled 2.5 million documents from the FinePDFs corpus, filtering for permissive licenses and high-quality renders across English, French, German, Italian, and Spanish.
- Synthetic Generation: They used a high-performing VLM (Qwen3.5-122B-A10B) to generate instruction-response pairs grounded in the document images. To ensure variety, they used "persona-based" prompting to vary the style and tone of instructions.
- Human Audit: Native linguists audited 3,000 examples to ensure the synthetic references were sound. They found a 90%+ acceptance rate, making the benchmark a reliable ground truth.
- Evaluation Protocol: They benchmarked 45 open-weight models (ranging from 1.7B to 235B parameters) and one commercial API (Gemini 3.8 Flash).
- Novel Metrics: For layout extraction, they moved away from standard geometric Intersection-over-Union (IoU) scores, which unfairly penalize models for tiny coordinate offsets on thin lines of text. Instead, they scored models based on the actual words covered by the predicted bounding boxes.
Datasets
The primary contribution is the MDU Dataset:
- Size: 23,146 examples.
- Languages: German (de), English (en), Spanish (es), French (fr), and Italian (it).
- Tasks: 13 tasks including Text/Markdown Extraction, Layout Extraction, Named Entity Recognition (NER), Classification, QA, Summarization, and Translation.
- Complexity: Includes both single-page and multi-page (2-5 pages) documents.
Key Insights and Ideas
- Scale is not a Silver Bullet: Surprisingly, the best-performing models were 27B parameter versions (Qwen3.6 and Qwen3.5), which outperformed massive models like the 235B Qwen3-VL. Model family architecture seems to be a better predictor of success than raw parameter count.
- The Persistent Language Gap: English remains the "easiest" language for models, while German is consistently the hardest. Critically, this gap does not shrink as models get larger; scale lifts all boats but keeps the distance between them the same.
- Skill Independence: A model’s ability to transcribe text (OCR) is largely uncorrelated with its ability to perform semantic extraction (like NER). This suggests that "document understanding" is not a monolithic skill but a collection of separable capabilities.
- The Multi-Page Penalty: Processing multiple pages is significantly harder for VLMs. Performance drops sharply on tasks like text extraction and translation when the model has to maintain coherence across page boundaries.
Key Results
- Benchmark Saturation: The suite is far from solved. The highest average score was only 69.0 (Qwen3.6 27B). Only eight out of 45 models managed to score above 65.
- Hardest Tasks: Layout Extraction and Entity Extraction proved to be the most difficult, with average scores across all models hovering around 34-35%.
- Commercial vs. Open-Weight: Gemini 3.8 Flash performed well (65.8 average, 7th overall) but was notably weaker at raw transcription tasks compared to specialized open-weight models.
Limitations and Future Work
- Linguistic Scope: The study is limited to five European languages using the Latin script. The authors acknowledge that these findings might not generalize to low-resource languages or non-Latin scripts (e.g., Arabic, Chinese).
- Synthetic Bias: Because a Qwen model was used to generate the benchmark, there is a risk of "self-preference" where Qwen-family models score higher due to stylistic similarities. The authors quantified this and found it accounts for about +7.8 points, which readers should keep in mind when viewing the leaderboard.
- Future Directions: The authors suggest that future work should focus on improving layout-grounded spatial reasoning and multi-page coherence, which remain the primary bottlenecks for current VLMs.
Summary updated