WeLayout: WeChat Layout Analysis System for the ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents

Mingliang Zhang, Zhen Cao, Juntao Liu, Liqiang Niu, Fandong Meng, Jie ZhouarXiv 2023 · DOI · arXiv

Abstract

The WeLayout system addresses the challenge of robust document layout analysis (DLA) in complex corporate documents. By combining DINO and YOLOv8 architectures with a novel "cell matching" refinement process that leverages PDF metadata, the system achieves high precision in identifying document elements like paragraphs and tables. The approach utilizes a multi-stage pipeline including document classification and an ensemble strategy optimized through Bayesian search. WeLayout secured first place in the ICDAR 2023 Competition with an mAP of 70.0, significantly outperforming competitors in handling unseen document categories.

Introduction

Automated document processing relies heavily on Document Layout Analysis (DLA)—the ability to categorize regions of a page such as paragraphs, tables, or headers. While standard OCR works well for plain text, corporate documents like financial reports, patents, and contracts feature complex, multi-column layouts and diverse visual elements that often confuse traditional systems. The ICDAR 2023 competition challenged researchers to build models robust across different document styles, specifically highlighting the difficulty of categories like "Others" which were not explicitly labeled in the training data.

Methods

The WeLayout system is a multi-stage pipeline designed for maximum precision, integrating classification, detection, and refinement stages.

The WeLayout system architecture, illustrating the pipeline from data augmentation to model ensemble and final result refinement.

1. Document Classification

Since different document types (e.g., patents vs. financial reports) possess distinct visual "grammars," the system uses a Vision Transformer (ViT) to classify incoming pages. This allows for category-specific fine-tuning, which the authors found significantly improved performance. For instance, YOLOv8's performance on manuals increased from 61.0 to 63.0 mAP when trained exclusively on that category.

2. Model Ensemble

The team combined two complementary state-of-the-art architectures:

  • DINO (Query-Based): A transformer-based detector using learnable queries to locate objects.
  • YOLOv8 (Anchor-Free): A high-speed detector that predicts objects per-pixel, avoiding the limitations of predefined anchor boxes.

3. Bounding Box Refinement (Cell Matching)

A standout feature of WeLayout is the use of the PDF text layer. Most digital PDFs contain "text cells" (coordinates of words/characters). The authors developed an algorithm to "snap" predicted bounding boxes to these precise boundaries. This post-processing step boosted DINO’s performance by 9.2 mAP points, demonstrating that deep learning models benefit greatly from ground-truth structural hints.

4. Weighted Box Fusion (WBF)

To merge predictions from DINO and YOLO, the system employs Weighted Box Fusion. The fusion parameters (weights and overlap thresholds) were optimized using Bayesian optimization (Tree-Structured Parzen Estimator), searching a hyperparameter space of nearly $10^{12}$ possibilities.

Datasets and Augmentation

The primary training data was DocLayNet (80,000+ annotated pages). To improve robustness, the authors harvested tables and figures from external datasets like IIIT-AR-13K and TNCR. Crucially, they generated 300,000 synthetic images by cropping elements and pasting them onto new backgrounds with varying column layouts.

Examples of document layouts handled by the system, ranging from single-column to multi-column structures.

Key Results

WeLayout achieved 1st place in the competition with an official mAP of 70.0. A key strength was its generalization; it outperformed the second-place team by 10 mAP points in the "Others" category. On an internal combined validation/test set, the full pipeline reached a peak performance of 91.0 mAP.

Conclusion

The success of WeLayout underscores the importance of combining diverse model architectures and leveraging existing metadata like PDF text layers. While the "Others" category remains a challenge due to the lack of dedicated training data, the system's classification-based approach provides a strong foundation for future robust layout analysis across even broader document types.

Summary updated

Abstract

In this paper, we introduce WeLayout, a novel system for segmenting the layout of corporate documents, which stands for WeChat Layout Analysis System. Our approach utilizes a sophisticated ensemble of DINO and YOLO models, specifically developed for the ICDAR 2023 Competition on Robust Layout Segmentation. Our method significantly surpasses the baseline, securing a top position on the leaderboard with a mAP of 70.0. To achieve this performance, we concentrated on enhancing various aspects of the task, such as dataset augmentation, model architecture, bounding box refinement, and model ensemble techniques. Additionally, we trained the data separately for each document category to ensure a higher mean submission score. We also developed an algorithm for cell matching to further improve our performance. To identify the optimal weights and IoU thresholds for our model ensemble, we employed a Bayesian optimization algorithm called the Tree-Structured Parzen Estimator. Our approach effectively demonstrates the benefits of combining query-based and anchor-free models for achieving robust layout segmentation in corporate documents.
BibTeX
@misc{zhang2023welayout,
  author = {Mingliang Zhang and Zhen Cao and Juntao Liu and Liqiang Niu and Fandong Meng and Jie Zhou},
  title = {{WeLayout: WeChat Layout Analysis System for the ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents}},
  year = {2023},
  eprint = {2305.06553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.06553},
}
Figure