DocBed: A Multi-Stage OCR Solution for Documents with Complex Layouts

Wenzhen Zhu, Negin Sokhandan, Guang Yang, Sujitha Martin, Suchitra Sathyanarayana

Abstract

Digitizing historical newspapers is a complex task due to their intricate, multi-column layouts and the presence of non-textual elements like advertisements and images. Standard Optical Character Recognition (OCR) systems often struggle with these structures, leading to fragmented or incorrectly ordered text. This paper presents DocBed, a multi-stage framework designed to segment complex layouts before OCR processing. By integrating state-of-the-art segmentation models and a novel separator detector, DocBed significantly enhances the accuracy of text extraction and reading order. Additionally, the authors introduce NewsNet7, a comprehensive dataset of 3,000 annotated historical newspaper pages to support further research in document layout analysis.

Introduction

Historical archives contain a wealth of information, yet much of it remains difficult to access digitally. Historical newspapers, in particular, feature chaotic layouts that defy the linear processing of traditional OCR engines. When an OCR engine encounters multi-column text interrupted by images or ads, it frequently merges columns or loses the intended human reading order. DocBed addresses this by treating layout segmentation as a critical precursor to character recognition.

The DocBed Framework

The DocBed pipeline replaces rigid, rule-based heuristics with deep learning models capable of generalizing across diverse printing styles. The framework operates in three primary stages: segmentation, separator detection, and post-processing.

The DocBed multi-stage pipeline, illustrating the flow from layout segmentation to OCR.

1. Layout Segmentation

The authors evaluated several architectures to classify pixels into categories such as "article body," "headline," and "advertisement":

  • Fully Convolutional Network (FCN): Established as a baseline.
  • Mask R-CNN: Utilized for instance segmentation with ResNet backbones.
  • Segmentation Transformer (SETR): Leverages global context-awareness to improve classification accuracy, treating segmentation as a sequence-to-sequence task.

2. Separator Detector (SepDet)

To ensure correct column splitting, the researchers developed SepDet, a specialized model focused on identifying "gutters"—the white spaces or lines separating text blocks. This component is vital for maintaining the logical flow of information.

3. Post-Processing

The pipeline concludes with geometric and heuristic post-processing. Geometric operations bridge small gaps between predicted blocks, while heuristics re-order these blocks from top-left to bottom-right to align with human reading patterns.

NewsNet7 Dataset

A significant contribution of this work is NewsNet7, a large-scale dataset comprising 3,000 scanned pages from 21 U.S. states. The dataset includes 169,508 manually annotated bounding boxes across seven categories: Header, Article Title, Article Body, Advertisement, Image, Table, and Other. This resource provides a robust benchmark for evaluating layout analysis models on real-world historical data.

Experimental Results

The researchers benchmarked DocBed using both NewsNet7 and the RDCL2019 dataset.

Qualitative comparison of layout segmentation results across different models: (a) Ground Truth, (b) FCN, (c) Mask R-CNN, (d) SETR, and (e) SepDet.
  • Segmentation Performance: The SETR model achieved a mean Intersection over Union (mIoU) of 70.02, outperforming the FCN baseline of 56.34. Its global attention mechanism proved particularly effective at identifying headers and titles based on their spatial relationships.
  • Read-Order Accuracy: The SepDet-geometric approach achieved a Read Order Accuracy (ROA) of 0.98 on the RDCL2019 dataset, demonstrating its effectiveness in preserving document structure.
  • OCR Enhancement: The SETR-heuristic approach resulted in the best Edit Distance (0.38) and Word Recall (0.94), confirming that superior layout understanding directly translates to higher OCR quality.
  • Efficiency: For production environments, Mask R-CNN with a ResNet-50-DC5 backbone offered the best balance, with an inference time of 0.08 seconds per image.

Conclusion

DocBed demonstrates that decoupling layout analysis from character recognition is essential for processing complex historical documents. By providing a robust multi-stage framework and the extensive NewsNet7 dataset, this work offers a path forward for more accurate digitization of global historical archives. Future research may explore graph-based methods to further refine reading order in non-linear, magazine-style layouts.

Summary updated

Abstract

Digitization of newspapers is of interest for many reasons including preservation of history, accessibility and search ability, etc. While digitization of documents such as scientific articles and magazines is prevalent in literature, one of the main challenges for digitization of newspaper lies in its complex layout (e.g. articles spanning multiple columns, text interrupted by images) analysis, which is necessary to preserve human read-order. This work provides a major breakthrough in the digitization of newspapers on three fronts: first, releasing a dataset of 3000 fully-annotated, real-world newspaper images from 21 different U.S. states representing an extensive variety of complex layouts for document layout analysis; second, proposing layout segmentation as a precursor to existing optical character recognition (OCR) engines, where multiple state-of-the-art image segmentation models and several post-processing methods are explored for document layout segmentation; third, providing a thorough and structured evaluation protocol for isolated layout segmentation and end-to-end OCR.
BibTeX
@inproceedings{zhu2022docbed,
  author = {Wenzhen Zhu and Negin Sokhandan and Guang Yang and Sujitha Martin and Suchitra Sathyanarayana},
  title = {{DocBed: A Multi-Stage OCR Solution for Documents with Complex Layouts}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21539},
}
Figure