2D Positional Embedding-based Transformer for Scene Text Recognition

Zobeir Raisi, Mohamed A. Naiel, Paul Fieguth, Steven Wardell, John ZelekJournal of Computational Vision and Imaging Systems 2021 · DOI

Abstract

Recent state-of-the-art scene text recognition methods are primarily based on Recurrent Neural Networks (RNNs). However, these methods require one-dimensional (1D) features and are not designed for recognizing irregular-text instances due to the loss of spatial information present in the original two-dimensional (2D) images. In this paper, we leverage a Transformer-based architecture for recognizing both regular and irregular text-in-the-wild images. The proposed method takes advantage of using a 2D positional encoder with the Transformer architecture to better preserve the spatial information of 2D image features than previous methods. The experiments on popular benchmarks, including the challenging COCO-Text dataset, demonstrate that the proposed scene text recognition method outperformed the state-of-the-art in most cases, especially on irregular-text recognition.

Introduction

Reading text "in the wild"—on street signs, product packaging, or billboards—is significantly harder than reading scanned documents. While standard OCR handles horizontal lines well, "irregular" text (curved, tilted, or distorted) poses a major challenge.

Most existing state-of-the-art methods rely on Recurrent Neural Networks (RNNs). To use an RNN, a 2D image must be flattened into a 1D sequence of features. This "flattening" process inherently destroys the spatial structure of the image. If a word is curved, its characters no longer follow a simple linear order, and 1D models struggle to piece them back together. While some researchers use "rectification" modules to straighten text before reading it, these add complexity and often fail on highly distorted images.

Methods

The authors propose a 2D Positional Embedding-based Transformer that preserves spatial relationships in both horizontal and vertical dimensions. The architecture consists of three main components:

  1. Feature Extraction (Backbone): A modified ResNet-31 CNN extracts high-level 2D feature maps from the input image.
  2. 2D Positional Encoding (PE): Since Transformers are permutation invariant, they require positional encoding. This paper uses a fixed 2D sinusoidal encoding that maps both the $x$ (horizontal) and $y$ (vertical) coordinates of every pixel in the feature map, ensuring the model knows exactly where each feature sits in 2D space.
  3. Modified Transformer Encoder/Decoder: The encoder uses multi-head self-attention to relate different parts of the image. Crucially, the standard Feed-Forward Network (FFN) is replaced with two layers of $1 \times 1$ convolutions, which are more robust for capturing image-based features.
Architecture of the 2D Positional Embedding-based Transformer.

Key Insights

  • Spatial Integrity: Irregular text recognition is fundamentally a spatial problem. By using 2D embeddings, the Transformer can "see" the geometric relationship between characters in a curve without needing to physically straighten the image first.
  • Rectification-Free: Unlike models like ASTER or RARE, this architecture does not use a Spatial Transformer Network (STN) to rectify text, suggesting that self-attention paired with 2D coordinates can learn geometric transformations implicitly.

Experimental Results

The model was evaluated on eight popular benchmarks, categorized into regular text (IIIT5k, SVT, ICDAR03, ICDAR13) and irregular text (ICDAR15, SVT-P, CUT80, COCO-Text).

The proposed method outperformed previous state-of-the-art (SOTA) models on seven out of eight datasets. On the SVT-P dataset (curved text), the model achieved 84.34% accuracy, a significant 4% lead over the previous best method. On the challenging COCO-Text dataset, it achieved 65.80% accuracy, outperforming ASTER's 60.70%. Qualitatively, the model correctly recognized highly distorted words where all other compared SOTA methods failed.

Limitations and Future Work

While the model is robust, it still struggles with heavy occlusion. The authors noted a failure case where "guide" was predicted as "guice" because the 'd' was partially blocked, though the ground truth itself was ambiguous in that instance.

A failure case where 'guide' is misrecognized as 'guice' due to occlusion.

Furthermore, the Transformer architecture is more computationally expensive than RNN-based frameworks during inference. Optimizing the speed for real-time applications and exploring even lighter backbones are primary goals for future work.

Summary updated

Abstract

Recent state-of-the-art scene text recognition methods are primarily based on Recurrent Neural Networks (RNNs), however, these methods require one-dimensional (1D) features and are not designed for recognizing irregular-text instances due to the loss of spatial information present in the original two-dimensional (2D) images. In this paper, we leverage a Transformer-based architecture for recognizing both regular and irregular text-in-the-wild images. The proposed method takes advantage of using a 2D positional encoder with the Transformer architecture to better preserve the spatial information of 2D image features than previous methods. The experiments on popular benchmarks, including the challenging COCO-Text dataset, demonstrate that the proposed scene text recognition method outperformed the state-of-the-art in most cases, especially on irregular-text recognition.
BibTeX
@article{raisi2021d,
  author = {Zobeir Raisi and Mohamed A. Naiel and Paul Fieguth and Steven Wardell and John Zelek},
  title = {{2D Positional Embedding-based Transformer for Scene Text Recognition}},
  journal = {Journal of Computational Vision and Imaging Systems},
  year = {2021},
  doi = {10.15353/jcvis.v6i1.3533},
}
Figure