TL;DR
Do-GOOD is a new benchmark designed to stress-test Visual Document Understanding (VDU) models against real-world "distribution shifts"—situations where test data looks different from training data. By systematically perturbing document images, text, and layouts, the researchers reveal that even state-of-the-art models like LayoutLMv3 are surprisingly brittle when faced with OCR errors, unusual backgrounds, or structural changes.
What Problem Does This Paper Address?
In the world of AI, we often assume that our training data and test data come from the same "distribution" (the i.i.d. assumption). However, in the wild, document AI models encounter messy realities: poor-quality scans, varied form layouts, and noisy OCR (Optical Character Recognition) output.
While pre-trained VDU models have achieved impressive scores on standard benchmarks, we don't actually know how robust they are to these "Out-of-Distribution" (OOD) scenarios. Existing OOD benchmarks for documents were either too narrow (focusing only on classification) or ignored the complex interplay between a document’s visual look, its text content, and its spatial layout. The authors created Do-GOOD to provide a fine-grained, multi-task evaluation suite to fill this gap.
Methods
The authors developed a systematic framework to induce distribution shifts across three modalities:
- Image Shifts: They disentangle document content from its background. They then replace the original background with either a "natural" image (from MSCOCO) or a "distorted" variant (using DocGeoNet) to simulate physical deformations and lighting issues.
- Text Shifts: To simulate OCR errors, they use BERT-Attack (replacing words with contextually similar but different ones) and Word Swap (homoglyphs, number swaps, or random character deletions).
- Layout Shifts: They manipulate the spatial structure by Merging (combining smaller bounding boxes into larger ones to test coarse-grained understanding) and Moving (shifting semantic entities to different locations on the page).
The study evaluates five top-tier VDU models (BROS, LiLT, and LayoutLM v1, v2, and v3) and tests whether standard OOD generalization algorithms (like Deep Coral and Mixup) actually help.
Datasets
The researchers introduced 9 new OOD datasets derived from three foundational VDU tasks:
- Information Extraction (FUNSD): Variants include
FUNSD-L(layout shifts),FUNSD-T(text shifts), andFUNSD-R/H(real-world and human-intervened shifts). - Document Classification (RVL-CDIP): Variants include
RVL-CDIP-T,RVL-CDIP-L, andRVL-CDIP-I(image shifts). - Visual Question Answering (DocVQA): The
DocVQA-Tvariant focuses on text-based distribution shifts.
Key Insights and Ideas
- The "Text-Layout" Dependency: A preliminary study revealed that for tasks like Information Extraction, models rely almost entirely on text and layout. Masking the image pixels entirely resulted in negligible performance drops, suggesting current models aren't "seeing" documents as much as they are "reading" their coordinates.
- The Brittle Nature of SOTA: Even the best models (LayoutLMv3) show a significant performance gap between In-Distribution (ID) and OOD settings. For example, on the RVL-CDIP classification task, accuracy can drop from ~95% to ~81% just by changing the layout.
- Vertical vs. Horizontal: In layout merging, the researchers found that "longitudinal" (vertical) merging hurts model accuracy significantly more than horizontal merging. This suggests models are more sensitive to the vertical flow of document information.
- OOD Algorithms Aren't a Silver Bullet: Common generalization techniques like Mixup improved performance in some classification scenarios but struggled with complex tasks like Question Answering, indicating that VDU-specific OOD solutions are still needed.
Key Results
- Performance Gaps: On the FUNSD (Information Extraction) task, LayoutLMv3 achieved an F1 score of 90.29 on standard data but plummeted to 57.88 on the real-world OOD-R dataset.
- Robustness Leaderboard: LayoutLMv3 emerged as the most robust model overall, particularly against text attacks, likely due to its unified text-and-image masking pre-training.
- Incremental Training: The authors demonstrated that adding even a small amount of OOD data (e.g., 20-50 samples) during training significantly boosts robustness, suggesting that "exposure" to diversity is key to model survival in the wild.
Limitations and Future Work
- Annotation Costs: Some of the most valuable OOD datasets (like FUNSD-R and FUNSD-H) required manual human annotation, which is expensive and difficult to scale.
- Algorithm Gap: The study shows that general-purpose OOD algorithms (Deep Coral, Mixup) don't fully solve the VDU problem. The authors suggest that future research should focus on VDU-specific OOD algorithms that better handle the structural "interactions" between text, vision, and layout.
- Scope: The current benchmark focuses on three core tasks; expanding this to more complex document types (like multi-page reports or handwritten notes) remains a future challenge.
Summary updated