Abstract
Most existing text reading benchmarks make it difficult to evaluate the performance of more advanced deep learning models in large vocabularies due to the limited amount of training data. To address this issue, we introduce a new large-scale text reading benchmark dataset named Chinese Street View Text (C-SVT) with 430,000 street view images, which is at least 14 times as large as the existing Chinese text reading benchmarks. To recognize Chinese text in the wild while keeping large-scale datasets labeling cost-effective, we propose to annotate one part of the C-SVT dataset (30,000 images) in locations and text labels as full annotations and add 400,000 more images, where only the corresponding text-of-interest in the regions is given as weak annotations. To exploit the rich information from the weakly annotated data, we design a text reading network in a partially supervised learning framework, which enables to localize and recognize text, learn from fully and weakly annotated data simultaneously. To localize the best matched text proposals from weakly labeled images, we propose an online proposal matching module incorporated in the whole model, spotting the keyword regions by sharing parameters for end-to-end training. Compared with fully supervised training algorithms, this model can improve the end-to-end recognition performance remarkably by 4.03% in F-score at the same labeling cost. The proposed model can also achieve state-of-the-art results on the ICDAR 2017-RCTW dataset, which demonstrates the effectiveness of the proposed partially supervised learning framework.
Introduction
Reading Chinese text in natural scenes is a formidable challenge in computer vision. Unlike Latin-based scripts, Chinese involves a massive vocabulary of thousands of characters with highly unbalanced frequency distributions. Furthermore, text in the wild appears in diverse orientations—horizontal, vertical, and even curved—and under varying lighting and occlusion conditions. A major bottleneck in developing robust deep learning models for this task has been the scarcity of large-scale, high-quality annotated data.
Existing benchmarks are often too small to capture the full complexity of Chinese text. The primary obstacle is the prohibitive cost of full annotation, which requires both precise bounding boxes (localization) and accurate transcriptions (recognition). To overcome this, researchers from Baidu and the University of Hong Kong have introduced the Chinese Street View Text (C-SVT) dataset and a novel partially supervised learning framework.
The C-SVT Dataset
The C-SVT dataset consists of 430,000 street view images, making it at least 14 times larger than previous Chinese text benchmarks. To manage annotation costs, the authors adopted a hybrid labeling strategy:
- Full Annotations: 30,000 images are labeled with both text locations (bounding boxes) and transcriptions.
- Weak Annotations: 400,000 images are labeled only with the "text-of-interest" (keywords) present in the image, without spatial information.
This approach is highly cost-effective. While fully annotating 30,000 images required approximately 6,600 man-hours, providing text labels for the remaining 400,000 images took only 960 man-hours.
Methodology: Partially Supervised Learning
The proposed framework is designed to learn from both fully and weakly annotated data simultaneously. It consists of four integrated components:
- Text Detection Branch: A ResNet-50 + FPN backbone generates potential text proposals.
- Perspective RoI Transform: This layer standardizes text regions of various orientations. Notably, to handle vertical Chinese text, the system rotates vertical proposals by 90 degrees before recognition, allowing a single model to process multiple orientations seamlessly.
- Text Recognition Branch: An attention-based encoder-decoder (CNN+RNN) transcribes features into characters.
- Online Proposal Matching (OPM) Module: This is the central innovation for leveraging weak labels. For images without bounding boxes, the OPM module calculates the similarity between the detector's proposals and the ground-truth keyword embeddings. It effectively "spots" the keyword regions during training, providing the necessary spatial supervision that would otherwise be missing.
The training process involves an initial stage using pseudo-labels from fully annotated data to warm up the OPM, followed by joint end-to-end training on the full dataset.
Results and Discussion
The partially supervised approach yields significant performance gains. On the C-SVT test set, the model (End2End-PSL) achieved a 4.03% improvement in F-score compared to a baseline trained only on fully supervised data. It also significantly reduced the Average Edit Distance (AED).
The model's robustness was further validated on the ICDAR 2017-RCTW benchmark, where it achieved a state-of-the-art F-score of 67.7% and an AED of 22.1. The OPM module demonstrated high precision (97%), although its recall (57%) remains an area for improvement, particularly when the detector fragments a single keyword into multiple proposals.
Conclusion
The C-SVT dataset and the partially supervised learning framework demonstrate that massive quantities of weakly labeled data can be more impactful than smaller sets of fully annotated data. By sharing parameters between recognition and proposal matching, the model learns robust features capable of localizing text with minimal spatial supervision. This work provides a scalable path forward for text recognition in complex scripts and large-vocabulary languages.
Summary updated