Abstract
Scene text recognition (STR) for Latin scripts has reached a high level of maturity, but Chinese STR remains a significant challenge due to the complexity and scale of the character set. This paper proposes a novel framework that incorporates a "Bag-of-Radicals" (BoR) prior to improve recognition accuracy. By decomposing characters into their constituent radicals and using them as both an embedding feature and a multi-task supervision signal, the model gains a deeper understanding of ideographic structures. Evaluated across six open-source Chinese STR datasets, the proposed method—combining Radical-Embedding (RE), a Character-Vector-Fusion-Module (CVFM), and multi-task training—demonstrates superior performance over existing state-of-the-art models.
Introduction
The field of Scene Text Recognition (STR) has seen rapid advancement, particularly for Latin-based languages. However, these successes do not translate directly to Chinese. The primary difficulty lies in the character set: while English relies on 26 letters, a standard Chinese character set includes nearly 6,000 frequently used characters, many of which possess intricate internal structures.
Previous research has attempted to use radicals—the fundamental building blocks of Chinese characters—to assist recognition. However, these methods typically treat radicals as a sequence, effectively "spelling out" each character. For a text line of 10 characters, the resulting radical sequence can exceed 50 tokens, leading to training difficulties and high inference latency. This paper addresses these issues by treating radicals as an unordered "bag," providing a more efficient structural prior.
Methodology
The proposed framework enhances a baseline STR model (GCAN) through three primary innovations:
1. Bag-of-Radicals (BoR) Representation
Instead of modeling the exact sequence of radicals, the authors decompose 5,941 Chinese characters into a set of 1,637 distinct basic radicals. Each character is represented by a 1,637-dimensional multi-hot vector. This "bag" approach captures the essential "ingredients" of a character without the computational burden of sequence prediction.
2. Character-Vector-Fusion-Module (CVFM)
To integrate this radical information into the neural network, the authors introduce the CVFM. This module fuses standard learnable character embeddings with Radical-Embeddings (RE). A learnable "scaler" is employed to balance the influence of the two embedding types, and a linear projection ensures they are mapped into a shared semantic space before concatenation.
3. Multi-task Radical Classification
To further sensitize the model to ideographic structures, a multi-task learning strategy is employed. During training, the model features a secondary classification head that predicts the presence of radicals within the input image. This acts as a supervision signal, forcing the visual encoder to learn features that correspond to the internal components of Chinese characters.
Experimental Results
The model was benchmarked against 10 state-of-the-art STR models across six diverse datasets: ICDAR-2019 ArT, CASIA-10k, ctw, LSVT, RCTW, and ReCTS. These datasets cover a wide range of challenges, including curved text, street views, and business signage.
Performance Gains
The integration of RE, CVFM, and multi-task training resulted in a 1.8% average improvement in word accuracy across all datasets. The most significant gains were observed in the most challenging environments:
- ICDAR-2019 ArT: 2.8% improvement (curved and distorted text).
- ctw (Chinese Text in the Wild): 2.3% improvement (complex street scenes).
Ablation Study
Ablation experiments confirmed that both the radical embeddings and the multi-task loss are critical. Removing either component led to a measurable drop in performance, suggesting that the model benefits both from the radical information as an input feature and as a training objective.
Discussion: The "Pupil" Analogy
The BoR approach mimics the cognitive process of learning Chinese. Much like a student (a "pupil") learns to recognize recurring components to distinguish between similar-looking characters, the model uses radicals to disambiguate complex ideographs. This structural prior is more robust than treating every character as an isolated, unique image. Furthermore, the CVFM module showed slight performance benefits even on Latin text, suggesting that the learnable fusion mechanism helps the model calibrate its attention more effectively across different scripts.
Limitations and Future Work
While the BoR prior significantly improves performance, certain challenges remain:
- Image Quality: Extremely blurry or low-resolution images make it difficult for the model to identify fine-grained radical details.
- Stylized Text: Artistic fonts and ancient calligraphic styles (e.g., Seal Script) often deviate from standard radical structures, leading to recognition errors.
- Low Contrast: Text that lacks clear contrast with its background remains a hurdle for the visual encoder.
Future research will explore the integration of stroke-level information and the development of more resilient visual encoders to handle the extreme variability of real-world scene text.
Conclusion
By introducing a Bag-of-Radicals prior, this work provides a computationally efficient and highly effective method for improving Chinese STR. The approach successfully leverages the unique ideographic nature of the Chinese writing system, narrowing the performance gap between Latin and non-Latin scene text recognition.
Summary updated