Reading Chinese in Natural Scenes with a Bag-of-Radicals Prior

Yongbin Liu, Qingjie Liu, Jiaxin Chen, Yunhong Wang

Abstract

Scene text recognition (STR) for Latin scripts has reached a high level of maturity, but Chinese STR remains a significant challenge due to the complexity and scale of the character set. This paper proposes a novel framework that incorporates a "Bag-of-Radicals" (BoR) prior to improve recognition accuracy. By decomposing characters into their constituent radicals and using them as both an embedding feature and a multi-task supervision signal, the model gains a deeper understanding of ideographic structures. Evaluated across six open-source Chinese STR datasets, the proposed method—combining Radical-Embedding (RE), a Character-Vector-Fusion-Module (CVFM), and multi-task training—demonstrates superior performance over existing state-of-the-art models.

Introduction

The field of Scene Text Recognition (STR) has seen rapid advancement, particularly for Latin-based languages. However, these successes do not translate directly to Chinese. The primary difficulty lies in the character set: while English relies on 26 letters, a standard Chinese character set includes nearly 6,000 frequently used characters, many of which possess intricate internal structures.

Previous research has attempted to use radicals—the fundamental building blocks of Chinese characters—to assist recognition. However, these methods typically treat radicals as a sequence, effectively "spelling out" each character. For a text line of 10 characters, the resulting radical sequence can exceed 50 tokens, leading to training difficulties and high inference latency. This paper addresses these issues by treating radicals as an unordered "bag," providing a more efficient structural prior.

Methodology

The proposed framework enhances a baseline STR model (GCAN) through three primary innovations:

1. Bag-of-Radicals (BoR) Representation

Instead of modeling the exact sequence of radicals, the authors decompose 5,941 Chinese characters into a set of 1,637 distinct basic radicals. Each character is represented by a 1,637-dimensional multi-hot vector. This "bag" approach captures the essential "ingredients" of a character without the computational burden of sequence prediction.

2. Character-Vector-Fusion-Module (CVFM)

To integrate this radical information into the neural network, the authors introduce the CVFM. This module fuses standard learnable character embeddings with Radical-Embeddings (RE). A learnable "scaler" is employed to balance the influence of the two embedding types, and a linear projection ensures they are mapped into a shared semantic space before concatenation.

3. Multi-task Radical Classification

To further sensitize the model to ideographic structures, a multi-task learning strategy is employed. During training, the model features a secondary classification head that predicts the presence of radicals within the input image. This acts as a supervision signal, forcing the visual encoder to learn features that correspond to the internal components of Chinese characters.

Experimental Results

The model was benchmarked against 10 state-of-the-art STR models across six diverse datasets: ICDAR-2019 ArT, CASIA-10k, ctw, LSVT, RCTW, and ReCTS. These datasets cover a wide range of challenges, including curved text, street views, and business signage.

Performance Gains

The integration of RE, CVFM, and multi-task training resulted in a 1.8% average improvement in word accuracy across all datasets. The most significant gains were observed in the most challenging environments:

  • ICDAR-2019 ArT: 2.8% improvement (curved and distorted text).
  • ctw (Chinese Text in the Wild): 2.3% improvement (complex street scenes).

Ablation Study

Ablation experiments confirmed that both the radical embeddings and the multi-task loss are critical. Removing either component led to a measurable drop in performance, suggesting that the model benefits both from the radical information as an input feature and as a training objective.

Discussion: The "Pupil" Analogy

The BoR approach mimics the cognitive process of learning Chinese. Much like a student (a "pupil") learns to recognize recurring components to distinguish between similar-looking characters, the model uses radicals to disambiguate complex ideographs. This structural prior is more robust than treating every character as an isolated, unique image. Furthermore, the CVFM module showed slight performance benefits even on Latin text, suggesting that the learnable fusion mechanism helps the model calibrate its attention more effectively across different scripts.

Limitations and Future Work

While the BoR prior significantly improves performance, certain challenges remain:

  • Image Quality: Extremely blurry or low-resolution images make it difficult for the model to identify fine-grained radical details.
  • Stylized Text: Artistic fonts and ancient calligraphic styles (e.g., Seal Script) often deviate from standard radical structures, leading to recognition errors.
  • Low Contrast: Text that lacks clear contrast with its background remains a hurdle for the visual encoder.

Future research will explore the integration of stroke-level information and the development of more resilient visual encoders to handle the extreme variability of real-world scene text.

Conclusion

By introducing a Bag-of-Radicals prior, this work provides a computationally efficient and highly effective method for improving Chinese STR. The approach successfully leverages the unique ideographic nature of the Chinese writing system, narrowing the performance gap between Latin and non-Latin scene text recognition.

Summary updated

Abstract

Scene text recognition (STR) on Latin datasets has been extensively studied in recent years, and state-of-the-art (SOTA) models often reach high accuracy. However, the performance on non-Latin transcripts, such as Chinese, is not satisfactory. In this paper, we collect six open-source Chinese STR datasets and evaluate a series of classic methods performing well on Latin datasets, finding a significant performance drop. To improve the performance on Chinese datasets, we propose a novel radical-embedding (RE) representation to utilize the ideographic descriptions of Chinese characters. The ideographic descriptions of Chinese characters are firstly converted to bags of radicals and then fused with learnable character embeddings by a character-vector-fusion-module (CVFM). In addition, we utilize a bag of radicals as supervision signals for multi-task training to improve the ideographic structure perception of our model. Experiments show performance of the model with RE + CVFM + multi-task training is superior compared with the baseline on six Chinese STR datasets. In addition, we utilize a bag of radicals as supervision signals for multi-task training to improve the ideographic structure perception of our model. Experiments show performance of the model with RE + CVFM + multi-task training is superior compared with the baseline on six Chinese STR datasets.
BibTeX
@inproceedings{liu2022reading,
  author = {Yongbin Liu and Qingjie Liu and Jiaxin Chen and Yunhong Wang},
  title = {{Reading Chinese in Natural Scenes with a Bag-of-Radicals Prior}},
  booktitle = {BMVC},
  year = {2022},
}
Figure