Multi-Lingual Optical Character Recognition System Using the Reinforcement Learning of Character Segmenter

Jaewoo Park, Eunji Lee, Yoonsik Kim, Isaac Kang, Hyung Il Koo, Nam Ik Cho

Abstract

While Latin-based Optical Character Recognition (OCR) is largely considered a solved problem, multi-lingual documents featuring mixed scripts—particularly Chinese, Japanese, and Korean (CJK)—present significant challenges due to massive character sets and complex internal structures. This paper proposes a modular OCR framework that utilizes Reinforcement Learning (RL) to train a character segmenter. By optimizing for final word-level edit distance rather than human-annotated boundaries, the system achieves superior performance on mixed-script documents compared to traditional segmentation-free models like Tesseract.

Introduction

The recognition of documents containing multiple languages, such as English mixed with Chinese or Korean, remains a difficult task for standard OCR engines. Modern "segmentation-free" architectures, often based on LSTM-CTC, excel at English but struggle with the thousands of unique characters in CJK scripts. These languages often exhibit varying intra-character spacing and complex structural overlaps, making it difficult for sequence models to discern clear boundaries without explicit segmentation. Traditional explicit segmentation methods, however, typically require massive amounts of hand-labeled data to define character boundaries, which is both tedious and expensive to produce.

Mixed-script word images illustrating the complexity of combining Korean, Chinese, and English characters in a single sequence.

Methodology

The proposed system employs a modular design consisting of three primary components:

  1. Segmenter: A neural network tasked with predicting optimal "cut" points to partition word images into individual characters.
  2. Switcher: A classifier that determines the language of each segmented sub-image, routing it to the appropriate recognizer.
  3. Multiple Recognizers: Specialized models tailored for English, Chinese, Korean, and special symbols.

Reinforcement Learning for Segmentation

The core innovation lies in the training of the Segmenter. Instead of using supervised learning with character boundary labels, the authors utilize the REINFORCE algorithm (a policy gradient method). The Segmenter acts as an agent making "cut" decisions, receiving a reward based on the final recognition accuracy (the inverse of the Levenshtein edit distance).

This approach ensures that the Segmenter learns to partition images in a way that maximizes the accuracy of the downstream recognizers, effectively adapting to their specific strengths and weaknesses. To overcome the "cold start" problem inherent in RL, the Segmenter undergoes a "warm-up" phase where it is pre-trained using simple, rule-based projection profiles to establish a reasonable starting policy.

Experimental Results

The system was evaluated on a combination of synthetic data (generated from Wikipedia) and real-world scanned documents. On mixed-script documents containing Chinese, English, and Korean, the proposed RL-based modular system achieved an accuracy of 90.21%, significantly outperforming Tesseract, which reached only 38.20%.

Qualitative examples of mixed-script words successfully processed by the modular OCR system, demonstrating the effectiveness of the RL-trained segmenter.

Ablation studies confirmed that the RL training was the critical factor; removing the REINFORCE optimization led to a nearly 50% drop in accuracy on mixed scripts. While the modular approach introduces higher computational complexity than single-stage models, its ability to handle massive character sets and complex overlaps makes it a robust solution for multi-lingual OCR in the wild.

Conclusion

By integrating Reinforcement Learning into a modular OCR pipeline, this research demonstrates a path forward for high-accuracy multi-lingual character recognition. The approach eliminates the need for tedious boundary labeling while providing the flexibility to handle diverse and complex scripts. Future work will focus on improving the computational efficiency of the multi-stage pipeline and scaling the system to support a broader range of global languages.

Summary updated

Abstract

In this article, we present a new multi-lingual Optical Character Recognition (OCR) system for scanned documents. In the case of Latin characters, current open source systems such as Tesseract provide very high accuracy. However, the accuracy of the multi-lingual documents, including Asian characters, is usually lower than that for Latin-only documents. For example, when the document is the mix of English, Chinese and/or Korean characters, the OCR accuracy is lowered than English-only because the character/text properties of Chinese and Korean are quite different from Latin-type characters. To tackle these problems, we propose a new framework using three neural blocks (a segmenter, a switcher, and multiple recognizers) and the reinforcement learning of the segmenter: The segmenter partitions a given word image into multiple character images, the switcher assigns a recognizer for each sub-image, and the recognizers perform the recognition of assigned sub-images. The training of recognizers and switcher can be considered traditional image classification tasks and we train them with a supervised learning method. However, the supervised learning of the segmenter has two critical drawbacks: Its objective function is sub-optimal and its training requires a large amount of annotation efforts. Thus, by adopting the REINFORCE algorithm, we train the segmenter so as to optimize the overall performance, i.e., we minimize the edit distance of final recognition results. Experimental results have shown that the proposed method significantly improves the performance for multi-lingual scripts and large character set languages without using character boundary labels.
BibTeX
@article{park2020multi,
  author = {Jaewoo Park and Eunji Lee and Yoonsik Kim and Isaac Kang and Hyung Il Koo and Nam Ik Cho},
  title = {{Multi-Lingual Optical Character Recognition System Using the Reinforcement Learning of Character Segmenter}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3025769},
}
Figure