Accurate Scene Text Detection through Border Semantics Awareness and Bootstrapping

Chuhui Xue, Shijian Lu, Fangneng Zhan

Introduction

Detecting text in natural scenes—such as street signs, storefronts, or posters—presents unique challenges compared to standard document scanning. This paper addresses three primary hurdles in scene text detection:

  1. Broken Detections: Long text lines are frequently fragmented into multiple boxes due to variations in lighting, font, or color.
  2. Inaccurate Localization: Standard regression methods often fail to pinpoint the exact ends of a text line, as pixels at the center of a long word are geometrically distant from the corners the model predicts.
  3. Limited Training Data: High-quality annotated datasets for scene text are relatively small, hindering the training of robust deep learning models.

Methods

The authors propose a multi-channel Fully Convolutional Network (FCN) enhanced by two novel strategies: bootstrapping-based data augmentation and semantics-aware text borders.

Bootstrapping-based Data Augmentation

Instead of relying solely on full annotated text boxes, the system samples random "subsections" of words or text lines. This approach effectively multiplies the training data and encourages the network to learn consistent features across different segments of a text line, preventing fragmentation during inference.

Semantics-Aware Text Borders

The model is trained to identify four distinct types of borders: two for the long sides (top/bottom) and two for the short sides (left/right).

  • Short-side borders are physically close to the corners of the text box. Regressing from these "end-cap" pixels significantly improves localization precision compared to regressing from the center of the word.
  • Long-side borders assist the model in distinguishing between closely stacked lines of text.

The final architecture employs a multi-task loss function that combines text classification, boundary regression, and border feature scoring.

Datasets and Evaluation

The model was evaluated on several industry-standard benchmarks:

  • MSRA-TD500: Multi-oriented Chinese and English text.
  • ICDAR2013: Horizontal English text.
  • ICDAR2017-RCTW: Large-scale Chinese text in the wild.
  • ICDAR2017-MLT: Multi-lingual dataset covering nine languages.
  • HUST-TR400: Supplementary data for MSRA-TD500.

Key Insights

  • Proximity Enhances Regression: Identifying short-side borders provides the model with pixels that have a reliable geometric relationship with the box's ends, reducing noise in corner prediction.
  • Consistency through Redundancy: Bootstrapping ensures that every segment of a text line produces a similar feature response, allowing the model to merge segments into a single, clean bounding box.
  • Efficiency: The authors found that using augmented subsections reduced training time from 56 hours to 36 hours, as shorter segments are easier for the network to learn initially.

Key Results

The proposed method achieved state-of-the-art or highly competitive results across all datasets. Notably, on MSRA-TD500, it achieved an 80.1 F-score, outperforming previous methods like SegLink and EAST.

The model's performance gap over baselines increased as the Intersection over Union (IoU) threshold became stricter, demonstrating the superior localization precision of the border semantics approach. Qualitative results show robustness against extreme perspective shifts and varied lighting.

Limitations and Future Work

Despite its success, the model faces challenges with:

  • Extreme Scale: Very large text can occasionally trigger false positives during Non-Maximum Suppression (NMS).
  • Ambiguous Annotations: Large gaps between words in a single line can lead to inconsistent grouping.
  • Orientation Confusion: Vertical text can sometimes be misinterpreted as a series of horizontal lines.

Future work may involve incorporating more explicit semantic information regarding text orientation to further improve detection accuracy.

Summary updated

BibTeX
@inproceedings{xue2018accurate,
  author = {Chuhui Xue and Shijian Lu and Fangneng Zhan},
  title = {{Accurate Scene Text Detection through Border Semantics Awareness and Bootstrapping}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=ByWPy5-OWB},
}
Figure