Abstract
Robust text reading from street view images provides valuable information for various applications. Performance improvement of existing methods in such a challenging scenario heavily relies on the amount of fully annotated training data, which is costly and inefficient to obtain. To scale up the amount of training data while keeping the labeling procedure cost-effective, this competition introduces a new challenge on Large-scale Street View Text with Partial Labeling (LSVT), providing 50,000 and 400,000 images in full and weak annotations, respectively. This competition aims to explore the abilities of state-of-the-art methods to detect and recognize text instances from large-scale street view images, closing the gap between research benchmarks and real applications. During the competition period, a total of 41 teams participated in the two proposed tasks with 132 valid submissions, i.e., text detection and end-to-end text spotting.
Introduction
Robust Reading Systems (RRS) are critical for the advancement of autonomous driving, automated translation, and urban navigation. However, these deep learning models are notoriously data-hungry. Traditionally, training requires "full annotation," where every text instance in an image is manually bounded by a polygon and transcribed—a process that is both slow and prohibitively expensive at scale.
While synthetic datasets like SynthText provide a starting point, they often fail to capture the "domain gap"—the messy, unpredictable reality of actual street views. The ICDAR 2019 RRC-LSVT competition addresses this by introducing a massive real-world dataset that utilizes partial labeling (weak annotations). This approach aims to bridge the gap between small, perfectly labeled research benchmarks and the vast, unlabeled data available in the real world.
The LSVT Dataset
The core contribution of this work is the Large-scale Street View Text (LSVT) dataset, which is approximately 14 times larger than previous robust reading benchmarks. Captured from diverse Chinese street scenes, including storefronts and landmarks, the dataset consists of:
- Total Images: 450,000.
- Fully Annotated: 30,000 training images where every text instance has a location and transcription.
- Weakly Annotated: 400,000 training images where only the "text-of-interest" (e.g., a primary store name) is transcribed, without location data.
- Testing: 20,000 images for leaderboard evaluation.
By focusing on "text-of-interest" for the weakly labeled set, the researchers prioritize practical utility, encouraging models to identify semantically meaningful text rather than every incidental character in the background.
Methodology
The competition was structured around two primary challenges:
- Task 1: Text Detection: Locating text instances using quadrilaterals or polygons to accommodate curved text.
- Task 2: End-to-End Text Spotting: Simultaneously detecting the location and recognizing the characters within the text.
Top-performing teams generally utilized powerful backbones like ResNet-152 or ResNeXt-101. Detection frameworks were largely dominated by variants of Mask R-CNN, often enhanced with Deformable Convolutions and Path Aggregation Networks (PANet) to handle irregular shapes. For recognition, teams typically employed CNNs for feature extraction combined with sequence modeling (Bi-directional GRUs or LSTMs) and either CTC (Connectionist Temporal Classification) or Attention-based decoders.
Results and Key Insights
The competition saw 41 teams deliver 132 valid submissions. The Tencent-DPPR team secured first place in both categories, achieving an H-mean of 86.42% in Task 1 and 60.97% in Task 2. Their success was largely attributed to model ensembles and multi-scale testing.
A key takeaway from the competition is the viability of scaling via weak labels. The results demonstrated that models can achieve high performance by augmenting a small "gold standard" fully labeled set with a massive "silver standard" weakly labeled set. This provides a more sustainable path for developing industry-scale AI.
Challenges and Future Work
Despite significant progress, the competition highlighted persistent difficulties in "wild" environments. The performance gap between detection (86.42%) and end-to-end spotting (60.97%) underscores the difficulty of accurate recognition under low contrast, partial occlusion, and varied fonts.
Specific areas for future research include:
- Vertical Text: Models still struggle to correctly group characters in vertical orientations, often splitting them incorrectly.
- Environmental Robustness: Low luminance and high-contrast backgrounds remain significant hurdles.
- End-to-End Integration: Moving beyond the "detect then recognize" two-stage pipeline toward tighter integration where recognition feedback improves detection accuracy.
- Semantic Exploitation: Leveraging the broader context of a scene (e.g., identifying a building as a restaurant) to aid in the recognition of ambiguous text.
Summary updated