Parallel Scale-wise Attention Network for Effective Scene Text Recognition

Usman Sajid, Michael Chow, Jin Zhang, Taejoon Kim, Guanghui Wang

TL;DR

This paper introduces a novel "Parallel Scale-wise Attention Network" (PSAN) for scene text recognition. By processing feature extraction and visual attention in parallel across multiple scales within the encoder, PSAN overcomes limitations of sequential, single-scale attention mechanisms, leading to improved accuracy on challenging scene text datasets.

What Problem Does This Paper Address?

Recognizing text in natural images (scene text recognition) is a challenging task due to variations in lighting, font styles, scale, and background clutter. While attention mechanisms have significantly improved performance by aligning text characters for recognition, existing state-of-the-art methods often apply attention sequentially after feature extraction, limiting its effectiveness. Specifically, these methods tend to rely on a single-scale feature representation, failing to fully leverage multi-scale information. This can lead to over-burdened decoders and error propagation. The paper aims to address these limitations by proposing a more efficient and effective attention mechanism that operates in parallel across multiple feature scales within the encoder.

Methods

The core innovation of this paper is the Parallel Scale-wise Attention Network (PSAN). Unlike previous methods that perform feature extraction (FE) and visual attention (VA) sequentially, PSAN integrates these processes in parallel within its encoder. The network architecture comprises three main components:

  • Text Encoder (EN): This is the heart of the proposed method. It takes the initial convolutional layer's output and processes it through three parallel multi-scale modules (S1, S2, S3). Each module performs:
  • Intra-Scale Processing: Within each scale module, residual structures (RS) are used for feature extraction, followed by a Visual Attention Block (VAB). The VAB applies attention independently to feature maps at that specific scale, helping to distinguish foreground from background. This is a departure from prior work that used more complex convolutional/deconvolutional layers for attention.
  • Repetitive Inter-scale Fusion: The scale modules also fuse information from different scales at regular intervals. This allows the network to build coordinated knowledge across scales. Higher-scale features are down-sampled to fuse with lower-scale features.
  • Merging Head (MH): This component takes the outputs from the three scale-modules (FS1, FS2, FS3), down-samples higher-scale features to match the resolution of the lowest scale (S3), concatenates them, and then uses a convolutional layer to adjust channels. The output is reshaped into K-dimensional vectors, where K is the number of character classes, and MaxLength represents the maximum text length.
  • Text Decoder (DE): This component is responsible solely for recognition, making it more focused. It uses a GRU cell to process the MH output sequentially and a classification layer with a softmax function to predict the character at each time step.

Datasets

The paper evaluates its proposed network on seven standard scene-text datasets:

  • Regular Datasets: IIIT-5K, IC03, IC13, SVT
  • Irregular Datasets: IC15, SVT-P, CUTE80

The network was trained using two synthetic datasets: Synth90k and SynthText.

Key Insights and Ideas

  • Parallelism is Key: The most significant insight is that performing feature extraction and visual attention in parallel across multiple scales within the encoder, rather than sequentially, dramatically improves performance. This allows attention to operate directly on richer, multi-scale representations.
  • Scale-wise Attention Matters: Applying visual attention independently at each scale (S1, S2, S3) helps the network gain a clearer understanding of foreground and background pixels at different levels of detail.
  • Inter-scale Fusion Enhances Context: Fusing information across scales allows the network to build a more comprehensive understanding of the scene text, especially for variations in text size and complexity.
  • Simpler Attention Mechanism is Effective: The proposed VAB, using a series of convolutions and a sigmoid function for segmentation maps, proves to be more effective than complex convolutional/deconvolutional attention mechanisms used in prior work.

Key Results

The proposed PSAN network achieves state-of-the-art or competitive results on most of the evaluated benchmarks.

  • It outperforms other methods on 4 out of 7 datasets.
  • On regular scene-text datasets (IIIT-5K and IC13), it shows an accuracy increase of 0.8% and 1.1% respectively.
  • On irregular scene-text datasets (IC15 and SVT-P), it improves accuracy by 1.4% and 1.0% respectively.
  • The paper also demonstrates the robustness of the proposed scheme against various image distortions (padding, scaling) in Table III, showing it to be more stable than previous methods like DAN.
  • Qualitative analysis (Fig. 7) shows that the VAB block significantly improves recognition accuracy, especially in cases with partial occlusion or similar-looking characters.

Limitations and Future Work

  • Inter-scale Fusion Simplicity: The authors acknowledge that their current inter-scale fusion technique is relatively simple. They plan to investigate more sophisticated fusion methods in the future.
  • Offline Recognition Focus: The current work focuses on offline recognition. The authors intend to explore efficiency and computational cost aspects for real-time and real-world applications in their future work.

Summary updated

BibTeX
@inproceedings{sajid2021parallel,
  author = {Usman Sajid and Michael Chow and Jin Zhang and Taejoon Kim and Guanghui Wang},
  title = {{Parallel Scale-wise Attention Network for Effective Scene Text Recognition}},
  booktitle = {IJCNN},
  year = {2021},
  doi = {10.1109/ijcnn52387.2021.9534223},
}
Figure