StrokeNet: Stroke Assisted and Hierarchical Graph Reasoning Networks

Lei Li, Kai Fan, Chun Yuan

TL;DR

This paper introduces StrokeNet, a novel approach for scene text detection that excels at identifying small, low-resolution, and arbitrarily shaped text by explicitly modeling fine-grained strokes. StrokeNet leverages a two-stage process: a Stroke Assisted Prediction Network (SAPN) to capture stroke details and a Hierarchical Relation Graph Network (HRGN) to reason about relationships between text and stroke elements, leading to state-of-the-art performance.

What Problem Does This Paper Address?

Detecting text in natural images (scene text detection) is a challenging problem, especially when dealing with texts that are extremely small, blurry, or have unusual shapes. Existing methods often struggle with these cases. Some methods focus on bounding boxes, which can be imprecise for arbitrary shapes. Others use segmentation but require complex post-processing. A key limitation is the difficulty in accurately capturing the fine-grained strokes that form characters, which are crucial for precise localization and for downstream tasks like text editing or translation. Furthermore, distinguishing between very close text instances remains a significant hurdle.

Methods

StrokeNet tackles these challenges with a two-pronged approach:

  • Stroke Assisted Prediction Network (SAPN): This module is designed to capture the intricate details of text strokes. It uses a backbone network (ResNet-50 with FPN) to predict text areas (TA) and text center areas (TCA), along with geometric attributes. Crucially, it includes a "Stroke-level Prediction" block that distills semantic information from text areas and filters it using multi-scale orthogonal convolutions and spatial attention to produce high-quality stroke cues. This allows for the detection of very small or low-resolution strokes.
  • Hierarchical Relation Graph Network (HRGN): After SAPN generates text- and stroke-level proposals, HRGN builds upon them. It constructs two types of graphs:
  • Isomorphic Stroke Graph: This graph models relationships between individual stroke nodes using an attention mechanism to aggregate information based on structure and content.
  • Heterogeneous Text Graph: This graph combines text-level nodes with stroke-level nodes. It uses a two-stage aggregation process: first, text-level nodes reason with their text neighbors, and then a "soft mask" mechanism selectively incorporates information from relevant stroke-level neighbors. This hierarchical reasoning helps in accurately splitting close text instances and grouping stroke information into complete text regions.

Datasets

  • SynthStroke: A novel dataset introduced by the authors, containing 800 thousand synthetic images with stroke-level segmentation masks and word-level bounding box annotations. It's designed to be more challenging than existing synthetic datasets like SynthText, featuring diverse fonts, orientations, and small text instances, making it ideal for pre-training text detectors.
  • Benchmark Datasets: StrokeNet is evaluated on several standard scene text detection benchmarks:
  • CTW-1500
    • Total-Text
    • MSRA-TD500
    • ICDAR 2015
    • ICDAR 2017 MLT
    • ICDAR 2019 MLT

Key Insights and Ideas

  • Strokes as the Fundamental Unit: The paper argues that explicitly modeling fine-grained strokes, rather than just bounding boxes or character segments, is key to accurately detecting challenging text instances, especially small and arbitrarily shaped ones.
  • Hierarchical Reasoning for Ambiguity: By building both stroke-level and text-level graphs and reasoning hierarchically, StrokeNet can effectively resolve ambiguities between closely spaced text instances and group stroke information into coherent text regions.
  • Synthetic Data for Pre-training: The introduction of SynthStroke highlights the importance of specialized synthetic datasets for pre-training models to handle specific challenges (like fine-grained strokes) that might be underrepresented in existing datasets.
  • Multi-task Learning for Strokes and Text: Combining stroke prediction with text-level prediction within a single framework (SAPN) allows the model to leverage complementary information for improved performance.

Key Results

StrokeNet achieves state-of-the-art performance on several benchmarks, particularly excelling in:

  • Close and Arbitrary-Shaped Text Detection: Demonstrates strong results on datasets like CTW-1500 and Total-Text, outperforming previous methods.
  • Small and Low-Resolution Text Detection: Achieves consistent and competitive performance on ICDAR 2015.
  • Multi-language Text Detection: Shows robust performance on datasets like MSRA-TD500, ICDAR 2017 MLT, and ICDAR 2019 MLT.
  • Pre-training Advantage: Using SynthStroke for pre-training significantly improves performance compared to using SynthText, demonstrating the value of the new dataset.
  • OCR Translation Application: The paper showcases an application of StrokeNet in an OCR translation pipeline, where stroke detection aids in accurate text recognition and inpainting.

Limitations and Future Work

  • Failure Cases: The authors acknowledge that StrokeNet can sometimes mistakenly detect edges of background objects as strokes, especially when these edges share similar features with text strokes. This is attributed to the sensitivity of orthogonal convolutions to edge features.
  • Future Work:
    • Further optimization to suppress background edges and improve stroke detection accuracy.
  • Exploring an end-to-end text detection framework that directly supports various downstream tasks like text recognition and text removal.

Summary updated

Abstract

Scene text detection is still a challenging task, as there may be extremely small or low-resolution strokes, and close or arbitrary-shaped texts. In this paper, StrokeNet is proposed to effectively detect the texts by capturing the fine-grained strokes, and infer structural relations between the hierarchical representation in the graph. Different from existing approaches that represent the text area by a series of points or rectangular boxes, we directly localize strokes of each text instance through Stroke Assisted Prediction Network (SAPN). Besides, Hierarchical Relation Graph Network (HRGN) is adopted to perform relational reasoning and predict the likelihood of linkages, effectively splitting the close text instances and grouping node classification results into arbitrary-shaped text region. We introduce a novel dataset with stroke-level annotations, namely SynthStroke, for offline pre-training of our model. Experiments on wide-ranging benchmarks verify the State-of-the-Art performance of our method. Our dataset and code will be available.
BibTeX
@article{li2021strokenet,
  author = {Lei Li and Kai Fan and Chun Yuan},
  title = {{StrokeNet: Stroke Assisted and Hierarchical Graph Reasoning Networks}},
  journal = {IEEE Trans. Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=3OLNpfcr1d},
}
Figure