Abstract
Document Layout Analysis (DLA) models often struggle with real-world document corruptions such as uneven lighting, motion blur, or physical warping. This paper introduces RoDLA, the first comprehensive robustness benchmark for DLA, featuring a taxonomy of 36 realistic document perturbations across 450,000 images. Alongside the benchmark, the authors propose the RoDLA model, which employs specialized attention and pooling mechanisms to achieve state-of-the-art robustness, outperforming existing models by up to 12.1% in mean Average Precision (mAP) on corrupted datasets.
Introduction
Document Layout Analysis (DLA) is a critical task in document processing, involving the identification and categorization of regions such as text, tables, and figures. While current state-of-the-art (SOTA) models perform exceptionally well on "clean" digital-born PDFs, their performance often degrades significantly when applied to real-world documents that have been scanned, photographed, or physically damaged. Factors like uneven lighting, camera vibration, and ink-bleeding introduce perturbations that standard models are not designed to handle. Until now, a standardized framework for measuring and improving the robustness of DLA models against such realistic corruptions has been lacking.
The RoDLA Benchmark
To address this gap, the researchers developed a comprehensive robustness benchmark consisting of three main components: a perturbation taxonomy, new evaluation metrics, and large-scale datasets.
Perturbation Taxonomy
The benchmark defines 12 types of document-specific perturbations, categorized into five groups: Spatial, Content, Inconsistency, Blur, and Noise. Each perturbation is applied at three severity levels (L1, L2, L3) to simulate varying degrees of document degradation.
Metrics and Datasets
The authors introduced two new metrics to quantify robustness:
- Mean Perturbation Effect (mPE): Measures the inherent difficulty of a perturbation by combining image quality scores with the performance drop of a baseline model.
- Mean Robustness Degradation (mRD): A normalized metric evaluating a model's robustness relative to perturbation difficulty. Lower scores indicate superior robustness.
The benchmark was applied to three major DLA datasets, creating "perturbed" versions totaling approximately 450,000 images: PubLayNet-P, DocLayNet-P, and M$^6$Doc-P.
The RoDLA Model Architecture
The proposed RoDLA model is built on the DINO detection framework and incorporates two key architectural enhancements to improve resilience:
- Channel-wise Attention (CA): This mechanism helps the model aggregate correlated features across channels while maintaining computational efficiency.
- Robust Encoder: By utilizing spatial-wise average pooling with varying dilation rates, the encoder effectively "ignores" fine-grained noise. This forces the model to focus on broader structural tokens, which are more reliable for layout analysis in corrupted images.
Experimental Results
The evaluation reveals that existing SOTA models are surprisingly fragile. For instance, the SwinDocSegmenter model experiences a 91% performance drop under heavy defocus blur. Interestingly, multimodal models (using text and layout) do not necessarily show better robustness than vision-only models when visual inputs are heavily corrupted.
RoDLA demonstrated significantly more stable performance across all severity levels:
- Performance Gains: RoDLA achieved mAP improvements of +3.8% on PubLayNet-P, +7.1% on DocLayNet-P, and +12.1% on M$^6$Doc-P compared to the next best models.
- Robustness: It obtained SOTA mRD scores, proving it degrades much more gracefully as image quality worsens.
- Efficiency: As a single-stage model, RoDLA is both more accurate and faster than traditional "two-stage" rectification pipelines that attempt to clean images before analysis.
Conclusion and Future Work
RoDLA sets a new standard for evaluating DLA robustness. While the current work focuses on vision-based models and individual perturbations, future research could extend the benchmark to multimodal frameworks and evaluate the effects of simultaneous, overlapping corruptions. Additionally, exploring robustness against adversarial content tampering remains an open challenge for the field.
Summary updated