Scene Text Recognition via Transformer

Xinjie Feng, Hongxun Yao, Yuankai Qi, Jun Zhang, Shengping ZhangarXiv 2020 · OpenReview

TL;DR

This paper introduces a novel scene text recognition method that completely bypasses the need for text rectification, a common bottleneck in existing approaches. By leveraging the full power of the Transformer architecture's attention mechanism, their method achieves state-of-the-art performance on both regular and arbitrary-shaped text recognition tasks, setting a new benchmark.

What Problem Does This Paper Address?

Recognizing text in images, especially when it's distorted, curved, or at an angle (arbitrary shapes), is a significant challenge in computer vision. Current leading methods often rely on a "rectification" step to straighten the text before recognition. However, this rectification process itself can introduce errors, particularly with severe distortions, leading to incorrect text identification. The paper argues that this rectification step is not only unnecessary but detrimental, and that focusing on spatial attention is the key to robust arbitrary-shaped text recognition.

Methods

The core of the proposed method is a Transformer-based architecture. Unlike previous Transformer applications in this domain that primarily used the decoder, this work utilizes both the encoder and decoder. Specifically:

  • Feature Extractor: A modified ResNet-101 is used to extract convolutional feature maps from the input image.
  • Transformer Encoder: These feature maps are then fed into the Transformer encoder. Crucially, the authors propose using these convolutional feature maps directly as word embeddings for the Transformer. This allows the encoder to fully leverage its powerful self-attention mechanism to capture spatial relationships within the text, even from a flattened representation.
  • Transformer Decoder: The decoder then generates the text sequence autoregressively, using the encoded representations.

The key innovation is the direct use of convolutional features as input to the Transformer encoder, enabling it to learn and recover spatial information without explicit rectification.

Datasets

The proposed method was evaluated on a comprehensive set of datasets:

  • Regular Text: IIIT5K, SVT
  • Irregular Text: IC15, SVTP, CUTE80

These datasets represent a range of challenges, including different text shapes, orientations, and levels of distortion.

Key Insights and Ideas

  • Rectification is Unnecessary: The most significant insight is that the complex and error-prone text rectification step can be entirely eliminated. The Transformer's attention mechanism is powerful enough to learn spatial relationships directly from the image features.
  • Full Transformer Power: By using both the encoder and decoder of the Transformer, the method effectively utilizes its global attention capabilities, which are superior to the local attention often found in RNN-based models.
  • Convolutional Features as Embeddings: Feeding convolutional feature maps directly into the Transformer encoder is a novel approach that allows the model to learn rich spatial representations.

Key Results

The proposed method demonstrates remarkable performance across various datasets:

  • Arbitrary Shape Text: Achieved state-of-the-art results, significantly outperforming previous methods. For instance, on the challenging CUTE dataset, it reached an accuracy of 99.3%, a substantial improvement over the previous best of 89.6%.
  • Regular Text: Despite being designed for arbitrary shapes, the method also performs very competitively on regular text datasets, often matching or exceeding existing top methods.
  • Generalization: The method shows strong generalization capabilities, performing well across diverse datasets with varying text characteristics.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Failure Cases: The method can still fail in extreme cases, such as with very difficult fonts, significant occlusion, or extremely low resolution.
  • Long Texts: Recognizing very long texts remains a challenge, potentially due to issues with resolution after resizing and the cascading nature of decoding.
  • Label Noise: The presence of label noise in test sets can also impact performance.

Future work could focus on addressing these limitations, potentially by exploring more robust feature extraction for challenging conditions or developing more advanced decoding strategies for longer sequences. The authors also plan to release their code, which will facilitate further research in this area.

Summary updated

Abstract

Scene text recognition with arbitrary shape is very challenging due to large variations in text shapes, fonts, colors, backgrounds, etc. Most state-of-the-art algorithms rectify the input image into the normalized image, then treat the recognition as a sequence prediction task. The bottleneck of such methods is the rectification, which will cause errors due to distortion perspective. In this paper, we find that the rectification is completely unnecessary. What all we need is the spatial attention. We therefore propose a simple but extremely effective scene text recognition method based on transformer [50]. Different from previous transformer based models [56,34], which just use the decoder of the transformer to decode the convolutional attention, the proposed method use a convolutional feature maps as word embedding input into transformer. In such a way, our method is able to make full use of the powerful attention mechanism of the transformer. Extensive experimental results show that the proposed method significantly outperforms state-of-the-art methods by a very large margin on both regular and irregular text datasets. On one of the most challenging CUTE dataset whose state-of-the-art prediction accuracy is 89.6%, our method achieves 99.3%, which is a pretty surprising result. We will release our source code and believe that our method will be a new benchmark of scene text recognition with arbitrary shapes.
BibTeX
@misc{feng2020scene,
  author = {Xinjie Feng and Hongxun Yao and Yuankai Qi and Jun Zhang and Shengping Zhang},
  title = {{Scene Text Recognition via Transformer}},
  year = {2020},
  eprint = {2003.08077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.08077},
}

Loading related content...

Figure