DocBed: A Multi-Stage OCR Solution for Documents with Complex Layouts

Wenzhen Zhu, Negin Sokhandan, Guang Yang, Sujitha Martin, Suchitra Sathyanarayana

Abstract

Digitization of newspapers is of interest for many reasons including preservation of history, accessibility and search ability, etc. While digitization of documents such as scientific articles and magazines is prevalent in literature, one of the main challenges for digitization of newspaper lies in its complex layout (e.g. articles spanning multiple columns, text interrupted by images) analysis, which is necessary to preserve human read-order. This work provides a major breakthrough in the digitization of newspapers on three fronts: first, releasing a dataset of 3000 fully-annotated, real-world newspaper images from 21 different U.S. states representing an extensive variety of complex layouts for document layout analysis; second, proposing layout segmentation as a precursor to existing optical character recognition (OCR) engines, where multiple state-of-the-art image segmentation models and several post-processing methods are explored for document layout segmentation; third, providing a thorough and structured evaluation protocol for isolated layout segmentation and end-to-end OCR.

Introduction

Digitizing historical newspapers presents a significant challenge for standard Optical Character Recognition (OCR) engines. Unlike modern documents with linear flows, historical newspapers are structured as complex grids with multiple columns, articles that jump across pages, and intrusive advertisements or images. Standard OCR engines often fail by reading straight across the page, mixing unrelated text and destroying the logical "read-order." DocBed addresses this by introducing an intelligent layout segmentation stage before the OCR process.

The DocBed Framework

The core innovation of DocBed is treating layout analysis as a necessary precursor to OCR. The pipeline consists of three distinct stages designed to transform a chaotic scan into structured, readable text.

The DocBed multi-stage pipeline: Layout Segmentation, Post-processing, and OCR.
  1. Layout Segmentation: Deep learning models identify regions such as headers, titles, body text, and advertisements. The study evaluated several architectures, including Fully Convolutional Networks (FCN), Mask R-CNN, and the Transformer-based SETR.
  2. Post-Processing: Raw segmentation masks are refined using geometric and heuristic algorithms to create "super boxes" that bridge small gaps and group related text blocks.
  3. OCR Execution: Instead of processing the entire page, the system sends individual super boxes to the OCR engine. This preserves the human read-order and improves text extraction accuracy.

NewsNet7: A New Benchmark for Layout Analysis

To train and evaluate these models, the authors released NewsNet7, a massive dataset of 3,000 scanned newspaper pages from 21 U.S. states. This dataset contains over 169,000 manually annotated instances across seven categories, providing the variety needed to handle the diverse layouts found in historical archives.

Experimental Results

The research compared several state-of-the-art segmentation models to determine which best handled the intricacies of newspaper layouts.

Comparative results of different segmentation models (FCN, Mask R-CNN, SETR, SepDet) against Ground Truth.

Key findings include:

  • Segmentation Accuracy: The SETR (Transformer-based) model achieved the highest mean Intersection over Union (mIoU) at 70.02%, benefiting from its ability to capture global context across the page.
  • OCR Improvement: Implementing the DocBed precursor significantly reduced the edit distance (error rate) from 0.63 (baseline) to 0.38 using the SETR-heuristic approach.
  • Read Order Preservation: The specialized "SepDet" model, which focuses on identifying column separators, achieved a near-perfect Read Order Accuracy (ROA) of 0.98.
  • Efficiency: While SETR was highly accurate, Mask R-CNN provided a superior balance for production environments, with an inference time of just 0.08s per page.

Conclusion

DocBed demonstrates that layout awareness is critical for the accurate digitization of complex historical documents. By providing both a robust multi-stage framework and the extensive NewsNet7 dataset, this work enables more reliable archival search and preservation. Future efforts will focus on end-to-end trainable networks and expanding the dataset to include non-English historical records.

Summary updated

Abstract

Digitization of newspapers is of interest for many reasons including preservation of history, accessibility and search ability, etc. While digitization of documents such as scientific articles and magazines is prevalent in literature, one of the main challenges for digitization of newspaper lies in its complex layout (e.g. articles spanning multiple columns, text interrupted by images) analysis, which is necessary to preserve human read-order. This work provides a major breakthrough in the digitization of newspapers on three fronts: first, releasing a dataset of 3000 fully-annotated, real-world newspaper images from 21 different U.S. states representing an extensive variety of complex layouts for document layout analysis; second, proposing layout segmentation as a precursor to existing optical character recognition (OCR) engines, where multiple state-of-the-art image segmentation models and several post-processing methods are explored for document layout segmentation; third, providing a thorough and structured evaluation protocol for isolated layout segmentation and end-to-end OCR.
BibTeX
@inproceedings{zhu2022docbed,
  author = {Wenzhen Zhu and Negin Sokhandan and Guang Yang and Sujitha Martin and Suchitra Sathyanarayana},
  title = {{DocBed: A Multi-Stage OCR Solution for Documents with Complex Layouts}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21539},
}
Figure