Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models

Jiabang He, Yi Hu, Lei Wang, Xing Xu, Ning Liu, Hui Liu, Heng Tao Shen

Abstract

Numerous pre-training techniques for visual document understanding (VDU) have recently shown substantial improvements in performance across a wide range of document tasks. However, these pre-trained VDU models cannot guarantee continued success when the distribution of test data differs from the distribution of training data. In this paper, to investigate how robust existing pre-trained VDU models are to various distribution shifts, we first develop an out-of-distribution (OOD) benchmark termed Do-GOOD for the fine-Grained analysis on Document image-related tasks specifically. The Do-GOOD benchmark defines the underlying mechanisms that result in different distribution shifts and contains 9 OOD datasets covering 3 VDU related tasks, e.g., document information extraction, classification and question answering. We then evaluate the robustness and perform a fine-grained analysis of 5 latest VDU pre-trained models and 2 typical OOD generalization algorithms on these OOD datasets. Results from the experiments demonstrate that there is a significant performance gap between the in-distribution (ID) and OOD settings for document images, and that fine-grained analysis of distribution shifts can reveal the brittle nature of existing pre-trained VDU models and OOD generalization algorithms.

Introduction

In the current landscape of AI, Visual Document Understanding (VDU) models have achieved remarkable success, often surpassing human-level performance on standard benchmarks. However, these successes typically rely on the i.i.d. (independent and identically distributed) assumption—that training and test data share the same underlying distribution. In real-world applications, this assumption frequently fails. Document AI systems must contend with poor-quality scans, varied form layouts, and noisy Optical Character Recognition (OCR) outputs.

While pre-trained models like LayoutLM have set new records, their robustness to "Out-of-Distribution" (OOD) scenarios remains largely unexplored. Existing OOD benchmarks often focus solely on image classification, ignoring the complex interplay between a document’s visual appearance, its textual content, and its spatial layout. To address this gap, the Do-GOOD benchmark provides a systematic, fine-grained evaluation suite for VDU models across multiple modalities and tasks.

The Do-GOOD Framework

The researchers developed a systematic framework to induce distribution shifts across three primary modalities: image, text, and layout. This allows for a granular understanding of which components of a document model are most susceptible to failure.

Overview of the five types of distribution shifts introduced in the Do-GOOD benchmark.

As illustrated above, the framework introduces five specific variants:

  1. Image Background Shifts: Disentangling document content from its background to test resilience against "natural" image backgrounds (e.g., MSCOCO) or "distorted" backgrounds (simulating physical deformations).
  2. Text Distribution Shifts: Simulating OCR errors and linguistic variations using BERT-Attack and Word Swap techniques (homoglyphs, number swaps, etc.).
  3. Layout Shifts: Manipulating spatial structure by merging bounding boxes to test coarse-grained understanding or moving semantic entities to different page locations.

Benchmarking Robustness

The Do-GOOD benchmark includes 9 new OOD datasets derived from three foundational VDU tasks: Information Extraction (using FUNSD), Document Classification (using RVL-CDIP), and Visual Question Answering (using DocVQA).

Experiments evaluating five top-tier VDU models (BROS, LiLT, and LayoutLM v1, v2, and v3) revealed a significant performance gap between In-Distribution (ID) and OOD settings. For instance, on the FUNSD information extraction task, the state-of-the-art LayoutLMv3 achieved an F1 score of 90.29 on standard data, but this plummeted to 57.88 when faced with real-world OOD-R datasets. This highlights the brittle nature of current models when faced with structural or textual noise.

Key Insights and Model Behavior

A critical finding of the study is the "Text-Layout Dependency." Many VDU models appear to rely almost exclusively on text and coordinate data rather than the actual visual pixels of the document.

Performance of LayoutLMv3 on NER tasks as the image region is increasingly masked.

As shown in the figure above, masking the image pixels—even up to 100%—resulted in negligible performance drops for Named Entity Recognition (NER). This suggests that current models are "reading" coordinates rather than truly "seeing" the document. Furthermore, the researchers found that "longitudinal" (vertical) layout merging hurts model accuracy significantly more than horizontal merging, indicating that models are highly sensitive to the vertical flow of document information.

While LayoutLMv3 emerged as the most robust model overall, standard OOD generalization algorithms like Mixup and Deep Coral were found to be insufficient. These general-purpose algorithms often struggled with the complex structural interactions inherent in VDU tasks, suggesting a need for domain-specific OOD solutions.

Conclusion

The Do-GOOD benchmark exposes the limitations of current pre-trained VDU models, demonstrating that high performance on standard datasets does not equate to real-world reliability. The significant performance drops observed across image, text, and layout shifts underscore the need for more robust pre-training objectives and VDU-specific OOD generalization algorithms. Future work will focus on scaling OOD data generation and developing models that can better integrate visual and structural information to overcome the current over-reliance on text coordinates.

Summary updated

Abstract

Numerous pre-training techniques for visual document understanding (VDU) have recently shown substantial improvements in performance across a wide range of document tasks. However, these pre-trained VDU models cannot guarantee continued success when the distribution of test data differs from the distribution of training data. In this paper, to investigate how robust existing pre-trained VDU models are to various distribution shifts, we first develop an out-of-distribution (OOD) benchmark termed Do-GOOD for the fine-Grained analysis on Document image-related tasks specifically. The Do-GOOD benchmark defines the underlying mechanisms that result in different distribution shifts and contains 9 OOD datasets covering 3 VDU related tasks, e.g., document information extraction, classification and question answering. We then evaluate the robustness and perform a fine-grained analysis of 5 latest VDU pre-trained models and 2 typical OOD generalization algorithms on these OOD datasets. Results from the experiments demonstrate that there is a significant performance gap between the in-distribution (ID) and OOD settings for document images, and that fine-grained analysis of distribution shifts can reveal the brittle nature of existing pre-trained VDU models and OOD generalization algorithms. The code and datasets for our Do-GOOD benchmark can be found at https://github.com/MAEHCM/Do-GOOD.
BibTeX
@inproceedings{he2023do,
  author = {Jiabang He and Yi Hu and Lei Wang and Xing Xu and Ning Liu and Hui Liu and Heng Tao Shen},
  title = {{Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3591670},
}
Figure