Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models

Jiabang He, Yi Hu, Lei Wang, Xing Xu, Ning Liu, Hui Liu, Heng Tao Shen

TL;DR

Do-GOOD is a new benchmark designed to stress-test Visual Document Understanding (VDU) models against real-world "distribution shifts"—situations where test data looks different from training data. By systematically perturbing document images, text, and layouts, the researchers reveal that even state-of-the-art models like LayoutLMv3 are surprisingly brittle when faced with OCR errors, unusual backgrounds, or structural changes.

What Problem Does This Paper Address?

In the world of AI, we often assume that our training data and test data come from the same "distribution" (the i.i.d. assumption). However, in the wild, document AI models encounter messy realities: poor-quality scans, varied form layouts, and noisy OCR (Optical Character Recognition) output.

While pre-trained VDU models have achieved impressive scores on standard benchmarks, we don't actually know how robust they are to these "Out-of-Distribution" (OOD) scenarios. Existing OOD benchmarks for documents were either too narrow (focusing only on classification) or ignored the complex interplay between a document’s visual look, its text content, and its spatial layout. The authors created Do-GOOD to provide a fine-grained, multi-task evaluation suite to fill this gap.

Methods

The authors developed a systematic framework to induce distribution shifts across three modalities:

  1. Image Shifts: They disentangle document content from its background. They then replace the original background with either a "natural" image (from MSCOCO) or a "distorted" variant (using DocGeoNet) to simulate physical deformations and lighting issues.
  2. Text Shifts: To simulate OCR errors, they use BERT-Attack (replacing words with contextually similar but different ones) and Word Swap (homoglyphs, number swaps, or random character deletions).
  3. Layout Shifts: They manipulate the spatial structure by Merging (combining smaller bounding boxes into larger ones to test coarse-grained understanding) and Moving (shifting semantic entities to different locations on the page).

The study evaluates five top-tier VDU models (BROS, LiLT, and LayoutLM v1, v2, and v3) and tests whether standard OOD generalization algorithms (like Deep Coral and Mixup) actually help.

Datasets

The researchers introduced 9 new OOD datasets derived from three foundational VDU tasks:

  • Information Extraction (FUNSD): Variants include FUNSD-L (layout shifts), FUNSD-T (text shifts), and FUNSD-R/H (real-world and human-intervened shifts).
  • Document Classification (RVL-CDIP): Variants include RVL-CDIP-T, RVL-CDIP-L, and RVL-CDIP-I (image shifts).
  • Visual Question Answering (DocVQA): The DocVQA-T variant focuses on text-based distribution shifts.

Key Insights and Ideas

  • The "Text-Layout" Dependency: A preliminary study revealed that for tasks like Information Extraction, models rely almost entirely on text and layout. Masking the image pixels entirely resulted in negligible performance drops, suggesting current models aren't "seeing" documents as much as they are "reading" their coordinates.
  • The Brittle Nature of SOTA: Even the best models (LayoutLMv3) show a significant performance gap between In-Distribution (ID) and OOD settings. For example, on the RVL-CDIP classification task, accuracy can drop from ~95% to ~81% just by changing the layout.
  • Vertical vs. Horizontal: In layout merging, the researchers found that "longitudinal" (vertical) merging hurts model accuracy significantly more than horizontal merging. This suggests models are more sensitive to the vertical flow of document information.
  • OOD Algorithms Aren't a Silver Bullet: Common generalization techniques like Mixup improved performance in some classification scenarios but struggled with complex tasks like Question Answering, indicating that VDU-specific OOD solutions are still needed.

Key Results

  • Performance Gaps: On the FUNSD (Information Extraction) task, LayoutLMv3 achieved an F1 score of 90.29 on standard data but plummeted to 57.88 on the real-world OOD-R dataset.
  • Robustness Leaderboard: LayoutLMv3 emerged as the most robust model overall, particularly against text attacks, likely due to its unified text-and-image masking pre-training.
  • Incremental Training: The authors demonstrated that adding even a small amount of OOD data (e.g., 20-50 samples) during training significantly boosts robustness, suggesting that "exposure" to diversity is key to model survival in the wild.

Limitations and Future Work

  • Annotation Costs: Some of the most valuable OOD datasets (like FUNSD-R and FUNSD-H) required manual human annotation, which is expensive and difficult to scale.
  • Algorithm Gap: The study shows that general-purpose OOD algorithms (Deep Coral, Mixup) don't fully solve the VDU problem. The authors suggest that future research should focus on VDU-specific OOD algorithms that better handle the structural "interactions" between text, vision, and layout.
  • Scope: The current benchmark focuses on three core tasks; expanding this to more complex document types (like multi-page reports or handwritten notes) remains a future challenge.

Summary updated

Abstract

Numerous pre-training techniques for visual document understanding (VDU) have recently shown substantial improvements in performance across a wide range of document tasks. However, these pre-trained VDU models cannot guarantee continued success when the distribution of test data differs from the distribution of training data. In this paper, to investigate how robust existing pre-trained VDU models are to various distribution shifts, we first develop an out-of-distribution (OOD) benchmark termed Do-GOOD for the fine-Grained analysis on Document image-related tasks specifically. The Do-GOOD benchmark defines the underlying mechanisms that result in different distribution shifts and contains 9 OOD datasets covering 3 VDU related tasks, e.g., document information extraction, classification and question answering. We then evaluate the robustness and perform a fine-grained analysis of 5 latest VDU pre-trained models and 2 typical OOD generalization algorithms on these OOD datasets. Results from the experiments demonstrate that there is a significant performance gap between the in-distribution (ID) and OOD settings for document images, and that fine-grained analysis of distribution shifts can reveal the brittle nature of existing pre-trained VDU models and OOD generalization algorithms. The code and datasets for our Do-GOOD benchmark can be found at https://github.com/MAEHCM/Do-GOOD.
BibTeX
@inproceedings{he2023do,
  author = {Jiabang He and Yi Hu and Lei Wang and Xing Xu and Ning Liu and Hui Liu and Heng Tao Shen},
  title = {{Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3591670},
}

Loading related content...

Figure