A robust and efficient algorithm for Chinese historical document analysis and recognition

Chongyu Liu, Cheng Jian, Jiarong Huang, Wentao Yang, Yongxin Shi, Qing Jiang, Lianwen JinNational Science Review 2023 · OpenReview

TL;DR

This paper introduces a novel, robust, and efficient algorithm for analyzing and recognizing Chinese historical documents, tackling challenges like image degradation and complex layouts. Their approach won first prize in a major competition, demonstrating significant improvements in both text detection and reading order prediction for historical texts.

What Problem Does This Paper Address?

Historical documents are invaluable repositories of human knowledge and culture, but their preservation and accessibility are hindered by the difficulty of digitizing them accurately. Traditional Optical Character Recognition (OCR) methods struggle with the unique challenges presented by historical documents, such as significant variations in page layouts, severe image degradation, diverse text fonts and scales, and text distortion. Existing benchmarks often fail to capture these real-world complexities, leading to algorithms that perform poorly in practical scenarios. This paper addresses this gap by developing an algorithm specifically designed to handle these challenging artifacts and by introducing a new benchmark, CHDAC, to better evaluate such systems.

Methods

The authors propose a four-component algorithm:

  • Novel Text Detection: They employ a Mask R-CNN framework enhanced with a HorNet-tiny backbone and a multi-oriented Region Proposal Network (RPN). The multi-oriented RPN is key, as it generates rotated region proposals by incorporating parameters that account for multi-directional text distributions, unlike standard RPNs. This is further refined by an iterative Region of Interest (RoI) head that performs iterative prediction with increasing Intersection over Union (IoU) thresholds, improving performance with fewer parameters than traditional cascade structures.
  • Text Recognition: A dual-path learning mechanism is introduced, combining a segmentation branch (predicting character location, bounding boxes, and classification) and a Connectionist Temporal Classification (CTC) branch. This hybrid approach leverages the strengths of both explicit and implicit segmentation methods to improve robustness, especially for long text recognition and degraded images. For training on CHDAC, which lacks character-level annotations, they use a weakly supervised training paradigm with pseudo-label updates.
  • End-to-End Connection: To seamlessly connect detection and recognition, they developed a detection point correction (DPC) thin plate spline (TPS) method. This method precisely crops text regions by correcting detection points and applying TPS transformation, ensuring the entire text is captured and distracting areas are eliminated.
  • Reading Order Prediction: A heuristic-based approach is used, which first rotates text boxes to horizontal if they have large skew angles. Then, it separates text into paragraphs based on Y-axis projection, aggregates text boxes within paragraphs into columns based on adjacency, and finally parses the internal reading order of each column using top-to-bottom and right-to-left rules.

Datasets

  • CHDAC Benchmark: A new benchmark named CHDAC was created for this competition. It comprises 8000 document images from various Chinese historical books (genealogy, medical books, Buddhist scriptures, local chronicles). The dataset features complex layouts, image degradation, diverse fonts/scales, and text distortion. It was split into training (4000 images), validation (1000 images), and test (3000 images) sets.
  • Other Benchmarks: For validating generalization ability, the authors also tested their method on MTH v2 and ICDAR 2019 HDRC CHINESE datasets.

Key Insights and Ideas

  • Multi-oriented RPN is Crucial: The ability to generate rotated region proposals is a significant improvement for detecting text in historical documents, which often have non-horizontal layouts.
  • Iterative RoI Head for Efficiency: Using an iterative RoI head with shared parameters offers a more parameter-efficient way to achieve high detection performance compared to cascaded approaches.
  • Dual-Path Recognition for Robustness: Combining segmentation-based and CTC-based recognition methods provides a more robust solution, mitigating errors caused by text degradation or long text sequences.
  • Heuristic Reading Order is Effective: A rule-based approach, while seemingly simple, proves effective for parsing the complex reading order of historical documents, especially when combined with initial text box orientation and paragraph/column segmentation.

Key Results

The proposed algorithm achieved outstanding results in the 2022 Greater Bay Area (Huangpu) International Algorithm Case Competition for Chinese Historical Document Recognition and Analysis Challenge (CHDAC):

  • Overall Performance: The algorithm won first prize out of 44 participants/teams, achieving an overall score of 97.95%, outperforming the second place by 2.05%.
  • Task 1 (Text Detection & Recognition): Achieved 91.43% Norm₁ on the final 3000 testing images, which was the best result.
  • Task 2 (Reading Order Prediction): Ranked in the top three with 1.61 Norm₂.
  • Efficiency: The inference speed was an average of 505 ms per image, and the model parameters took 187 MB in total, both ranking in the top three.
  • Floating-point Operations: The detection model achieved 806.65 G FLOPS, and the recognition model achieved 57.10 G FLOPS.
  • Generalization: The method demonstrated strong performance on other historical document benchmarks (MTH v2 and ICDAR 2019 HDRC CHINESE), outperforming previous methods.

Limitations and Future Work

The authors acknowledge several areas for future research:

  • Long-Tailed Distribution: The inherent long-tailed distribution of character categories in historical documents, where some characters appear very rarely, poses a significant challenge for recognition.
  • Generalization to Diverse Layouts: While their method shows good generalization, further work is needed to handle even more complex and diverse layouts and text styles.
  • Multi-lingual Documents: Extending the approach to handle multi-lingual historical documents (e.g., Chinese, English, Arabic) in a unified model is a significant challenge.
  • Zero-shot/Few-shot Learning: Developing methods for zero-shot or few-shot learning is crucial due to the difficulty of collecting sufficient labeled training samples for all categories.
  • Self-supervised Pre-training: Leveraging the abundance of unlabeled historical document data through self-supervised learning is a promising avenue for improving performance.
  • Global Platform: The ultimate goal is to construct a unified, general platform for automatic information processing of historical documents, which requires addressing the multi-lingual text challenge.

Summary updated

BibTeX
@article{liu2023robust,
  author = {Chongyu Liu and Cheng Jian and Jiarong Huang and Wentao Yang and Yongxin Shi and Qing Jiang and Lianwen Jin},
  title = {{A robust and efficient algorithm for Chinese historical document analysis and recognition}},
  journal = {National Science Review},
  year = {2023},
  url = {https://openreview.net/forum?id=nrhNgQNSB8},
}
Figure