ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling - RRC-LSVT

Yipeng Sun, Zihan Ni, Chee Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, Chee Seng Chan, Lianwen Jin

Abstract

Robust text reading from street view images provides valuable information for various applications. Performance improvement of existing methods in such a challenging scenario heavily relies on the amount of fully annotated training data, which is costly and inefficient to obtain. To scale up the amount of training data while keeping the labeling procedure cost-effective, this competition introduces a new challenge on Large-scale Street View Text with Partial Labeling (LSVT), providing 50,000 and 400,000 images in full and weak annotations, respectively. This competition aims to explore the abilities of state-of-the-art methods to detect and recognize text instances from large-scale street view images, closing the gap between research benchmarks and real applications. During the competition period, a total of 41 teams participated in the two proposed tasks with 132 valid submissions, i.e., text detection and end-to-end text spotting.

Introduction

Robust Reading Systems (RRS) are critical for the advancement of autonomous driving, automated translation, and urban navigation. However, these deep learning models are notoriously data-hungry. Traditionally, training requires "full annotation," where every text instance in an image is manually bounded by a polygon and transcribed—a process that is both slow and prohibitively expensive at scale.

While synthetic datasets like SynthText provide a starting point, they often fail to capture the "domain gap"—the messy, unpredictable reality of actual street views. The ICDAR 2019 RRC-LSVT competition addresses this by introducing a massive real-world dataset that utilizes partial labeling (weak annotations). This approach aims to bridge the gap between small, perfectly labeled research benchmarks and the vast, unlabeled data available in the real world.

The LSVT Dataset

The core contribution of this work is the Large-scale Street View Text (LSVT) dataset, which is approximately 14 times larger than previous robust reading benchmarks. Captured from diverse Chinese street scenes, including storefronts and landmarks, the dataset consists of:

  • Total Images: 450,000.
  • Fully Annotated: 30,000 training images where every text instance has a location and transcription.
  • Weakly Annotated: 400,000 training images where only the "text-of-interest" (e.g., a primary store name) is transcribed, without location data.
  • Testing: 20,000 images for leaderboard evaluation.
Sample images from the LSVT dataset illustrating diverse text instances and annotations.

By focusing on "text-of-interest" for the weakly labeled set, the researchers prioritize practical utility, encouraging models to identify semantically meaningful text rather than every incidental character in the background.

Methodology

The competition was structured around two primary challenges:

  1. Task 1: Text Detection: Locating text instances using quadrilaterals or polygons to accommodate curved text.
  2. Task 2: End-to-End Text Spotting: Simultaneously detecting the location and recognizing the characters within the text.

Top-performing teams generally utilized powerful backbones like ResNet-152 or ResNeXt-101. Detection frameworks were largely dominated by variants of Mask R-CNN, often enhanced with Deformable Convolutions and Path Aggregation Networks (PANet) to handle irregular shapes. For recognition, teams typically employed CNNs for feature extraction combined with sequence modeling (Bi-directional GRUs or LSTMs) and either CTC (Connectionist Temporal Classification) or Attention-based decoders.

Results and Key Insights

The competition saw 41 teams deliver 132 valid submissions. The Tencent-DPPR team secured first place in both categories, achieving an H-mean of 86.42% in Task 1 and 60.97% in Task 2. Their success was largely attributed to model ensembles and multi-scale testing.

A key takeaway from the competition is the viability of scaling via weak labels. The results demonstrated that models can achieve high performance by augmenting a small "gold standard" fully labeled set with a massive "silver standard" weakly labeled set. This provides a more sustainable path for developing industry-scale AI.

Challenges and Future Work

Despite significant progress, the competition highlighted persistent difficulties in "wild" environments. The performance gap between detection (86.42%) and end-to-end spotting (60.97%) underscores the difficulty of accurate recognition under low contrast, partial occlusion, and varied fonts.

Examples of complex text layouts in the LSVT dataset, including vertical and multi-line text.

Specific areas for future research include:

  • Vertical Text: Models still struggle to correctly group characters in vertical orientations, often splitting them incorrectly.
  • Environmental Robustness: Low luminance and high-contrast backgrounds remain significant hurdles.
  • End-to-End Integration: Moving beyond the "detect then recognize" two-stage pipeline toward tighter integration where recognition feedback improves detection accuracy.
  • Semantic Exploitation: Leveraging the broader context of a scene (e.g., identifying a building as a restaurant) to aid in the recognition of ambiguous text.

Summary updated

Abstract

Robust text reading from street view images provides valuable information for various applications. Performance improvement of existing methods in such a challenging scenario heavily relies on the amount of fully annotated training data, which is costly and in-efficient to obtain. To scale up the amount of training data while keeping the labeling procedure cost-effective, this competition introduces a new challenge on Large-scale Street View Text with Partial Labeling (LSVT), providing 50, 000 and 400, 000 images in full and weak annotations, respectively. This competition aims to explore the abilities of state-of-the-art methods to detect and recognize text instances from large-scale street view images, closing the gap between research benchmarks and real applications. During the competition period, a total of 41 teams participated in the two proposed tasks with 132 valid submissions, i.e., text detection and end-to-end text spotting. This paper includes dataset descriptions, task definitions, evaluation protocols and results summaries of the ICDAR 2019-LSVT challenge.
BibTeX
@inproceedings{sun2019icdar,
  author = {Yipeng Sun and Zihan Ni and Chee Kheng Chng and Yuliang Liu and Canjie Luo and Chun Chet Ng and Junyu Han and Errui Ding and Jingtuo Liu and Dimosthenis Karatzas and Chee Seng Chan and Lianwen Jin},
  title = {{ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling - RRC-LSVT}},
  booktitle = {ICDAR},
  year = {2019},
  doi = {10.1109/icdar.2019.00250},
}
Figure