RoDLA: Benchmarking the Robustness of Document Layout Analysis Models

Yufan Chen, Jiaming Zhang, Kunyu Peng, Junwei Zheng, Ruiping Liu, Philip Torr, Rainer Stiefelhagen

Abstract

Document Layout Analysis (DLA) models often struggle with real-world document corruptions such as uneven lighting, motion blur, or physical warping. This paper introduces RoDLA, the first comprehensive robustness benchmark for DLA, featuring a taxonomy of 36 realistic document perturbations across 450,000 images. Alongside the benchmark, the authors propose the RoDLA model, which employs specialized attention and pooling mechanisms to achieve state-of-the-art robustness, outperforming existing models by up to 12.1% in mean Average Precision (mAP) on corrupted datasets.

Introduction

Document Layout Analysis (DLA) is a critical task in document processing, involving the identification and categorization of regions such as text, tables, and figures. While current state-of-the-art (SOTA) models perform exceptionally well on "clean" digital-born PDFs, their performance often degrades significantly when applied to real-world documents that have been scanned, photographed, or physically damaged. Factors like uneven lighting, camera vibration, and ink-bleeding introduce perturbations that standard models are not designed to handle. Until now, a standardized framework for measuring and improving the robustness of DLA models against such realistic corruptions has been lacking.

The RoDLA Benchmark

To address this gap, the researchers developed a comprehensive robustness benchmark consisting of three main components: a perturbation taxonomy, new evaluation metrics, and large-scale datasets.

Perturbation Taxonomy

The benchmark defines 12 types of document-specific perturbations, categorized into five groups: Spatial, Content, Inconsistency, Blur, and Noise. Each perturbation is applied at three severity levels (L1, L2, L3) to simulate varying degrees of document degradation.

Visualizations of the 12 document perturbations (P1-P12) used in the RoDLA benchmark.

Metrics and Datasets

The authors introduced two new metrics to quantify robustness:

  • Mean Perturbation Effect (mPE): Measures the inherent difficulty of a perturbation by combining image quality scores with the performance drop of a baseline model.
  • Mean Robustness Degradation (mRD): A normalized metric evaluating a model's robustness relative to perturbation difficulty. Lower scores indicate superior robustness.

The benchmark was applied to three major DLA datasets, creating "perturbed" versions totaling approximately 450,000 images: PubLayNet-P, DocLayNet-P, and M$^6$Doc-P.

The RoDLA Model Architecture

The proposed RoDLA model is built on the DINO detection framework and incorporates two key architectural enhancements to improve resilience:

  1. Channel-wise Attention (CA): This mechanism helps the model aggregate correlated features across channels while maintaining computational efficiency.
  2. Robust Encoder: By utilizing spatial-wise average pooling with varying dilation rates, the encoder effectively "ignores" fine-grained noise. This forces the model to focus on broader structural tokens, which are more reliable for layout analysis in corrupted images.

Experimental Results

The evaluation reveals that existing SOTA models are surprisingly fragile. For instance, the SwinDocSegmenter model experiences a 91% performance drop under heavy defocus blur. Interestingly, multimodal models (using text and layout) do not necessarily show better robustness than vision-only models when visual inputs are heavily corrupted.

Overview of the RoDLA framework: (a) Taxonomy of perturbations, (b) mRD results across groups, and (c) mAP performance across severity levels.

RoDLA demonstrated significantly more stable performance across all severity levels:

  • Performance Gains: RoDLA achieved mAP improvements of +3.8% on PubLayNet-P, +7.1% on DocLayNet-P, and +12.1% on M$^6$Doc-P compared to the next best models.
  • Robustness: It obtained SOTA mRD scores, proving it degrades much more gracefully as image quality worsens.
  • Efficiency: As a single-stage model, RoDLA is both more accurate and faster than traditional "two-stage" rectification pipelines that attempt to clean images before analysis.

Conclusion and Future Work

RoDLA sets a new standard for evaluating DLA robustness. While the current work focuses on vision-based models and individual perturbations, future research could extend the benchmark to multimodal frameworks and evaluate the effects of simultaneous, overlapping corruptions. Additionally, exploring robustness against adversarial content tampering remains an open challenge for the field.

Summary updated

Abstract

Before developing a Document Layout Analysis (DLA) model in real-world applications, conducting comprehensive robustness testing is essential. However, the robustness of DLA models remains underexplored in the literature. To address this, we are the first to introduce a robustness benchmark for DLA models, which includes 450K document images of three datasets. To cover realistic corruptions, we propose a perturbation taxonomy with 12 common document perturbations with 3 severity levels inspired by realworld document processing. Additionally, to better understand document perturbation impacts, we propose two metrics, Mean Perturbation Effect (mPE) for perturbation assessment and Mean Robustness Degradation (mRD) for robustness evaluation. Furthermore, we introduce a self-titled model, i.e., Robust Document Layout Analyzer (RoDLA), which improves attention mechanisms to boost extraction of robust features. Experiments on the proposed benchmarks (PubLayNet-P, DocLayNet-P, andM 6 Doc-P) demonstrate that RoDLA obtains state-of-the-art mRD scores of 115.7, 135.4, and 150.4, respectively. Compared to previous methods, RoDLA achieves notable improvements in mAP of +3.8%, +7.1% and +12.1%, respectively.
BibTeX
@inproceedings{chen2024rodla,
  author = {Yufan Chen and Jiaming Zhang and Kunyu Peng and Junwei Zheng and Ruiping Liu and Philip Torr and Rainer Stiefelhagen},
  title = {{RoDLA: Benchmarking the Robustness of Document Layout Analysis Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure