Reading Chinese in Natural Scenes with a Bag-of-Radicals Prior

Yongbin Liu, Qingjie Liu, Jiaxin Chen, Yunhong Wang

TL;DR

This paper introduces a "Bag-of-Radicals" (BoR) prior to improve Scene Text Recognition (STR) for Chinese characters. By decomposing complex characters into their constituent radicals and using them as both an embedding feature and a multi-task supervision signal, the authors significantly narrow the performance gap between Latin and Chinese text recognition in "in-the-wild" images.

What Problem Does This Paper Address?

While state-of-the-art models for Latin scene text recognition (like English) have reached high levels of accuracy, Chinese STR remains a formidable challenge. The primary hurdle is the sheer scale and complexity of the character set: while English uses 26 letters, a standard Chinese charset contains nearly 6,000 common characters, many with intricate, overlapping structures.

Previous attempts to use radicals (the building blocks of Chinese characters) often treated them as sequences—essentially "spelling out" a character. However, for a string of 10-20 characters, the resulting radical sequence becomes prohibitively long (50-100 tokens), making the model slow and difficult to train. This paper addresses the need for a more efficient way to incorporate structural knowledge into deep learning models without the computational overhead of long-sequence modeling.

Methods

The authors propose a framework that treats radicals as a "bag" (an unordered set) rather than a sequence. Their approach consists of three main components:

  1. Bag-of-Radicals (BoR) Representation: They utilize a dictionary to decompose 5,941 Chinese characters into 1,637 distinct basic radicals. Each character is represented by a 1,637-dimensional multi-hot vector indicating which radicals are present.
  2. Character-Vector-Fusion-Module (CVFM): This module fuses standard learnable character embeddings with the new Radical-Embeddings (RE). It uses a learnable "scaler" to adjust the influence of the embeddings and a linear projection to ensure both representations exist in the same semantic space before being concatenated.
  3. Multi-task Radical Classification: During training, the model doesn't just predict the character; it also features a secondary "head" that predicts which radicals are present in the image. This acts as a supervision signal, forcing the encoder to learn features that are sensitive to the internal ideographic structure of the characters.

Datasets

The authors benchmarked their approach against 10 well-known STR models across six large-scale open-source Chinese datasets:

  • ICDAR-2019 ArT: Focuses on curved and complex shapes.
  • CASIA-10k: A diverse set of scene text detections.
  • ctw (Chinese Text in the Wild): Primarily street view images.
  • LSVT (Large-scale Street View Text): A massive dataset of 213k+ cropped images.
  • RCTW (ICDAR-2017): A classic competition dataset.
  • ReCTS (Reading Chinese Text on Signboard): Focused on business signage.

Key Insights and Ideas

  • Structure as a Prior: The "aha moment" here is that you don't need to know the order of radicals to identify a character; knowing the ingredients (the bag) is often enough to disambiguate similar-looking characters.
  • Efficiency over Sequence: By moving from a sequence-based radical model to a bag-based one, the authors avoid the "vanishing gradient" and high latency issues associated with very long output strings.
  • The "Pupil" Analogy: The method mimics how children learn Chinese—by recognizing recurring components (radicals) rather than memorizing every character as a unique, unrelated picture.
  • Cross-Language Benefits: Interestingly, the CVFM module slightly improved performance on Latin text as well, likely because the learnable scaler helped the model better calibrate its attention on simpler character sets.

Key Results

The proposed model, which adds RE + CVFM + multi-task training to a baseline (GCAN), outperformed all other tested SOTA models.

  • Accuracy Boost: The model achieved a 1.8% average improvement in word accuracy across all transcripts compared to the baseline.
  • Robustness: The largest gains were seen in challenging datasets like ArT (2.8% gain) and ctw (2.3% gain), which contain highly distorted or "wild" text.
  • Ablation Success: Removing the radical embeddings or the multi-task loss led to significant performance drops, validating that both the feature (RE) and the goal (BCE loss) are necessary for success.

Limitations and Future Work

Despite the improvements, the authors identified several persistent failure modes:

  • Low Resolution and Blur: When images are extremely blurry, the model struggles to distinguish fine-grained radical details.
  • Artistic/Ancient Fonts: Highly decorated or ancient calligraphic styles (like "Seal Script") deviate too far from the standard radical structures the model was trained on.
  • Low Contrast: Text that blends into the background (e.g., red text on a dark red background) remains difficult to segment and recognize.

Future work may involve integrating stroke-level information or developing more robust visual encoders specifically designed to handle the low-contrast and high-blur environments of real-world street scenes.

Summary updated

Abstract

Scene text recognition (STR) on Latin datasets has been extensively studied in recent years, and state-of-the-art (SOTA) models often reach high accuracy. However, the performance on non-Latin transcripts, such as Chinese, is not satisfactory. In this paper, we collect six open-source Chinese STR datasets and evaluate a series of classic methods performing well on Latin datasets, finding a significant performance drop. To improve the performance on Chinese datasets, we propose a novel radical-embedding (RE) representation to utilize the ideographic descriptions of Chinese characters. The ideographic descriptions of Chinese characters are firstly converted to bags of radicals and then fused with learnable character embeddings by a character-vector-fusion-module (CVFM). In addition, we utilize a bag of radicals as supervision signals for multi-task training to improve the ideographic structure perception of our model. Experiments show performance of the model with RE + CVFM + multi-task training is superior compared with the baseline on six Chinese STR datasets. In addition, we utilize a bag of radicals as supervision signals for multi-task training to improve the ideographic structure perception of our model. Experiments show performance of the model with RE + CVFM + multi-task training is superior compared with the baseline on six Chinese STR datasets.
BibTeX
@inproceedings{liu2022reading,
  author = {Yongbin Liu and Qingjie Liu and Jiaxin Chen and Yunhong Wang},
  title = {{Reading Chinese in Natural Scenes with a Bag-of-Radicals Prior}},
  booktitle = {BMVC},
  year = {2022},
}

Loading related content...

Figure