Primitive Representation Learning for Scene Text Recognition

Ruijie Yan, Liangrui Peng, Shanyu Xiao, Gang YaoCVPR 2021 · DOI

Abstract

Scene text recognition (STR) faces significant challenges due to the diverse orientations and shapes of text in natural images. Traditional methods, such as CTC-based or attention-based models, often struggle with irregular text or suffer from misalignment issues. This paper introduces the Primitive REpresentation learning Network (PREN), a novel framework that treats image features as nodes in a graph to learn global "primitive representations." By moving away from sequential processing, PREN achieves superior accuracy and efficiency, particularly on irregular and multi-oriented text datasets.

Introduction

Reading text in the wild—on signs, labels, or billboards—is a complex task because text is rarely perfectly horizontal or uniformly typed. It can be curved, tilted, or vertically oriented. Existing state-of-the-art methods typically follow two paradigms:

  1. CTC-based methods (e.g., CRNN): These treat text as a 1D sequence, which fails when the 1D assumption is violated by vertical or highly curved text.
  2. Attention-based Encoder-Decoders: These process images step-by-step but are prone to "misalignment" or "drift," where errors early in the sequence propagate and cause the model to lose track of character positions.
Comparison of (a) CTC-based, (b) Attention-based, and (c) the proposed PREN approach.

The authors of PREN argue that these sequential approaches lack a holistic, global understanding of the text's visual structure.

Methodology

The core innovation of PREN is the Primitive Representation. Instead of slicing the image into a sequence, the model treats the feature map as a collection of nodes in an undirected graph, learning a set of global base vectors.

Learning the "Primitives"

The architecture extracts $n$ primitive vectors from feature maps using two distinct aggregators:

  • Pooling Aggregator: Captures intrinsic, shared structural information across images via global average pooling.
  • Weighted Aggregator: Dynamically focuses on specific regions (e.g., character boundaries) by generating sample-specific heatmaps.
Detailed architecture of the PREN framework, highlighting the feature extraction and primitive representation learning modules.

Visual Text Representations and GCNs

These primitives are processed by Graph Convolutional Networks (GCNs), which transform abstract base vectors into high-level "visual text representations." Each resulting vector corresponds to a character position in the word.

Model Variants: PREN and PREN2D

  • PREN: A streamlined version designed for parallel decoding. By avoiding recursive character-by-character decoding, it achieves high speeds without the risk of error accumulation.
  • PREN2D: A more robust variant that integrates these global visual representations into a 2D attention-based Transformer. This "global guidance" effectively resolves the misalignment problem common in standard attention models.

Evaluation and Results

The models were trained on standard synthetic datasets (MJSynth and SynthText) and evaluated across various benchmarks:

  • Regular English: IIIT5k, SVT, IC03, IC13.
  • Irregular English: IC15, SVT-Perspective (SVTP), CUTE80 (curved text).
  • Chinese Scene Text: A subset of the RCTW dataset, featuring complex, multi-oriented characters.

Key Performance Metrics

  • Accuracy: PREN2D set new benchmarks on irregular datasets, reaching 83.0% on IC15 and 91.7% on CUTE80.
  • Efficiency: The basic PREN model is exceptionally fast, processing images in 22.7ms on a Tesla V100—outperforming standard CTC-based models in both speed and accuracy.
  • Orientation Robustness: On vertical Chinese text, PREN2D achieved 85.0% accuracy, a massive improvement over the 59.1% achieved by a standard CNN-LSTM-CTC baseline.

Conclusion

PREN demonstrates that breaking the sequential bottleneck is key to robust scene text recognition. By utilizing GCNs and global aggregation, the model treats text as a structured whole rather than a simple sequence. While the number of primitives $n$ requires careful tuning and PREN2D introduces additional computational overhead, the framework provides a powerful, parallelizable alternative to traditional STR methods.

Summary updated

BibTeX
@inproceedings{yan2021primitive,
  author = {Ruijie Yan and Liangrui Peng and Shanyu Xiao and Gang Yao},
  title = {{Primitive Representation Learning for Scene Text Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00035},
}
Figure