Accurate Scene Text Detection Through Border Semantics Awareness and Bootstrapping

Chuhui Xue, Shijian Lu, Fangneng Zhan

TL;DR

This paper introduces two novel techniques to improve scene text detection: a bootstrapping method that samples text "subsections" to handle limited data and improve prediction consistency, and a semantics-aware border detection technique that uses four distinct border types to achieve much more precise localization. The resulting model significantly reduces "broken" detections and excels at accurately bounding long, complex text lines.

What Problem Does This Paper Address?

Detecting text in natural scenes (like street signs or posters) is notoriously difficult compared to document scanning. The authors identify three primary hurdles:

  1. Broken Detections: Long text lines are often mistakenly detected as multiple fragmented boxes because of variations in lighting, font, or color across the line.
  2. Inaccurate Localization: Standard regression methods often struggle to find the exact ends of a text line. This happens because the pixels at the center of a long word are geometrically far from the corners (vertices) the model is trying to predict.
  3. Limited Training Data: High-quality annotated datasets for scene text are relatively small, making it hard to train robust deep learning models.

Methods

The authors propose a multi-channel Fully Convolutional Network (FCN) enhanced by two key strategies:

  • Bootstrapping-based Data Augmentation: Instead of just using the full annotated text box, the system repeatedly samples random "subsections" of a word or text line. This effectively multiplies the amount of training data and forces the network to learn consistent features across different parts of a text line, which prevents the "broken box" problem during inference.
  • Semantics-Aware Text Borders: Rather than treating all text pixels the same, the model is trained to identify four specific types of borders: two for the long sides (top/bottom) and two for the short sides (left/right).
  • Short-side borders are crucial because they are physically close to the corners of the text box. By regressing from these "end-cap" pixels rather than the center of the word, the model achieves much higher localization precision.
  • Long-side borders help the model distinguish between neighboring lines of text that are stacked closely together.

The final architecture uses a multi-task loss function combining classification (is it text?), regression (where are the boundaries?), and border feature scores.

Datasets

The model was evaluated on several standard industry benchmarks:

  • MSRA-TD500: Focuses on multi-oriented text (Chinese and English).
  • ICDAR2013: A classic dataset for horizontal English text.
  • ICDAR2017-RCTW: A large-scale dataset for Chinese text in the wild.
  • ICDAR2017-MLT: A multi-lingual dataset covering nine different languages.
  • HUST-TR400: Used to supplement training data for MSRA-TD500.

Key Insights and Ideas

  • Proximity Matters for Regression: One of the "aha" moments is the realization that regressing a box corner from a pixel in the middle of a long sentence is mathematically "noisy." By identifying the short-side borders, the model identifies the specific pixels that have the most reliable geometric relationship with the box's ends.
  • Consistency through Redundancy: The bootstrapping method doesn't just add data; it ensures that every segment of a text line produces a similar feature response. This "consistency" is what allows the model to merge segments into a single, clean bounding box rather than a series of fragments.
  • Faster Convergence: Surprisingly, the authors found that using augmented "subsections" actually made the model converge faster (36 hours vs. 56 hours). They hypothesize that shorter, simpler segments are easier for the network to learn initially than full, complex text lines.

Key Results

The proposed method achieved state-of-the-art or highly competitive results across all tested datasets:

  • MSRA-TD500: Achieved an 80.1 F-score, outperforming previous methods like SegLink and EAST.
  • Precision at High IoU: The model's performance gap over the baseline increased as the requirements for accuracy (IoU threshold) became stricter. This proves the "border semantics" approach significantly improves localization precision.
  • Robustness: Qualitative results show the model successfully handling extreme perspective shifts, varied lighting, and very long text lines that typically cause other models to fail.

Limitations and Future Work

Despite its strengths, the authors noted a few failure cases:

  • Extreme Scale: Very large text can sometimes lead to false positives due to the Non-Maximum Suppression (NMS) process.
  • Ambiguous Annotations: If a long text line has massive gaps between words, the model might struggle to decide whether to treat it as one line or several, especially if the ground-truth labels are inconsistent.
  • Vertical vs. Horizontal: In some cases, the model can become confused by vertical text that could also be interpreted as a series of single-character horizontal lines.

The authors suggest that incorporating more explicit semantic information about text orientation could be a promising next step.

Summary updated

Abstract

This paper presents a scene text detection technique that exploits bootstrapping and text border semantics for accurate localization of texts in scenes. A novel bootstrapping technique is designed which samples multiple ‘subsections’ of a word or text line and accordingly relieves the constraint of limited training data effectively. At the same time, the repeated sampling of text ‘subsections’ improves the consistency of the predicted text feature maps which is critical in predicting a single complete instead of multiple broken boxes for long words or text lines. In addition, a semantics-aware text border detection technique is designed which produces four types of text border segments for each scene text. With semantics-aware text borders, scene texts can be localized more accurately by regressing text pixels around the ends of words or text lines instead of all text pixels which often leads to inaccurate localization while dealing with long words or text lines. Extensive experiments demonstrate the effectiveness of the proposed techniques, and superior performance is obtained over several public datasets, e.g. 80.1 f-score for the MSRA-TD500, 67.1 f-score for the ICDAR2017-RCTW, etc.
BibTeX
@inproceedings{xue2018accurate,
  author = {Chuhui Xue and Shijian Lu and Fangneng Zhan},
  title = {{Accurate Scene Text Detection Through Border Semantics Awareness and Bootstrapping}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=ByWPy5-OWB},
}
Figure