Self-Attention Based Text Knowledge Mining for Text Detection

Qi Wan, Haoqin Ji, Linlin ShenCVPR 2021 · OpenReview

Abstract

Pre-trained models play an important role in deep learning based text detectors. However, most methods ignore the gap between natural images and scene text images and directly apply ImageNet for pre-training. To address such a problem, some of them firstly pre-train the model using a large amount of synthetic data and then fine-tune it on target datasets, which is task-specific and has limited generalization capability. In this paper, we focus on providing general pre-trained models for text detectors. Considering the importance of exploring text contents for text detection, we propose STKM (Self-attention based Text Knowledge Mining), which consists of a CNN Encoder and a Self-attention Decoder, to learn general prior knowledge for text detection from SynthText. Given only image level text labels, Self-attention Decoder directly decodes features extracted from CNN Encoder to texts without requirement of detection, which guides the CNN backbone to explicitly learn discriminative semantic representations ignored by previous approaches. After that, the text knowledge learned by the backbone can be transferred to various text detectors to significantly improve their detection performance.

Addressing the Pre-training Gap

Most modern scene text detectors rely on backbones pre-trained on ImageNet. While ImageNet is the gold standard for general computer vision, there is a massive "domain gap" between identifying general objects (like animals or vehicles) and identifying text in the wild.

Comparison of pre-training strategies highlighting the domain and knowledge gaps.

Current workarounds often involve pre-training on large synthetic datasets (like SynthText) using a specific detector. However, this approach suffers from two major flaws:

  1. Task Specificity: Weights trained for one detector (e.g., box-based) often fail to transfer effectively to another (e.g., segmentation-based), creating a "knowledge gap."
  2. Ignoring Content: Most detectors treat text as "texture" or "blobs," ignoring the semantic character-level information that distinguishes text from visually similar background noise like fences or windows.

The STKM Framework

The authors introduce STKM (Self-attention based Text Knowledge Mining), a framework designed to teach a neural network what text is (its semantic content) rather than just where it is located. By training a backbone to recognize text sequences using only image-level labels, they create a "text-aware" model that significantly boosts downstream performance.

The STKM architecture showing the CNN Encoder, ASPM, and Self-attention Decoder.

1. The "Knowledge Mining" Task

Instead of training the model to draw boxes, STKM performs multi-word recognition. Given an image, the model must output a sequence of all characters present. This requires only image-level text labels, which are significantly easier to obtain than pixel-level annotations.

2. Key Architectural Components

  • CNN Encoder: Utilizes a standard ResNet backbone with a Feature Pyramid Network (FPN) to extract multi-scale features.
  • Adaptive Spatial Position Encoder (ASPM): Injects spatial coordinate information back into flattened features, allowing the Transformer-style decoder to maintain awareness of character positions.
  • Self-attention Decoder: A cascaded architecture that decodes visual features into an ordered sequence of characters.

3. Ordered Word Sequence Generation (OTSG)

To structure the recognition task, the OTSG algorithm sorts words by ASCII order and uses special tokens like SOS (start-of-sequence) and WOS (word-separation). This forces the backbone to learn discriminative semantic representations to "read" the entire image. Once pre-training is complete, the decoder is discarded, and the "High-Level Knowledge" (HLK) in the backbone is transferred to a downstream detector.

Key Results and Generalization

STKM pre-training consistently outperforms standard ImageNet pre-training across multiple benchmarks:

  • EAST Detector: Achieved a 5.89% higher F-measure on ICDAR2015 and a 10.86% boost on MSRA-TD500.
  • PSENet Detector: Saw improvements of 5.64% on ICDAR2015 and 3.84% on TotalText.

The core insight is that a model that knows how to read is naturally better at finding text. By learning character dependencies (e.g., that a 'u' often follows a 'q'), the backbone becomes robust against visual noise. Furthermore, because the task is recognition-based, the weights are highly universal, transferring successfully to both regression-based (EAST) and segmentation-based (PSENet) architectures.

Conclusion

STKM provides a powerful, general-purpose pre-training strategy for text detection. By mining semantic text knowledge from synthetic data, it bridges the gap between general image recognition and specialized text detection tasks, offering significant performance gains without adding inference-time complexity.

Pre-trained model is available at: https://github.com/CVI-SZU/STKM

Summary updated

Abstract

Pre-trained models play an important role in deep learning based text detectors. However, most methods ignore the gap between natural images and scene text images and directly apply ImageNet for pre-training. To address such a problem, some of them firstly pre-train the model using a large amount of synthetic data and then fine-tune it on target datasets, which is task-specific and has limited generalization capability. In this paper, we focus on providing general pre-trained models for text detectors. Considering the importance of exploring text contents for text detection, we propose STKM (Self-attention based Text Knowledge Mining), which consists of a CNN Encoder and a Self-attention Decoder, to learn general prior knowledge for text detection from SynthText. Given only image level text labels, Self-attention Decoder directly decodes features extracted from CNN Encoder to texts without requirement of detection, which guides the CNN backbone to explicitly learn discriminative semantic representations ignored by previous approaches. After that, the text knowledge learned by the backbone can be transferred to various text detectors to significantly improve their detection performance (e.g., 5.89% higher F-measure for EAST on ICDAR15 dataset) without bells and whistles. Pre-trained model is available at: https://github.com/CVI-SZU/STKM
BibTeX
@inproceedings{wan2021self,
  author = {Qi Wan and Haoqin Ji and Linlin Shen},
  title = {{Self-Attention Based Text Knowledge Mining for Text Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=rk6_uxio-Mo},
}
Figure