TL;DR
This paper introduces a novel, efficient, and robust attention network for scene text recognition, particularly excelling with irregular text shapes. By directly connecting a CNN encoder to an attention-based decoder and incorporating a "holistic representation" of the image, the model achieves state-of-the-art performance while being significantly faster than previous RNN-based methods and requiring only word-level annotations.
What Problem Does This Paper Address?
Recognizing text in natural images (scene text recognition) is a challenging task, especially when the text is distorted, curved, or arbitrarily oriented. While existing methods have made progress, they often rely on complex shape rectification, multi-direction encoding, character detection, or recurrent neural networks (RNNs). These approaches can be difficult to train, computationally expensive, and may struggle with the inherent 2D nature of irregular text by forcing it into a 1D sequence representation. The authors aim to simplify this process by proposing a more direct and efficient approach.
Methods
The core of the proposed method is a CNN encoder-decoder architecture inspired by the Transformer but adapted for image input.
- CNN Encoder: A modified ResNet34 is used to extract features. Crucially, it produces two outputs:
- A 2D convolutional feature map: This captures local spatial information.
- A 1D holistic representation: This is a compressed, global representation of the entire text image, designed to provide context to the decoder.
- Attention-Based Decoder: This decoder is non-recurrent, allowing for parallel training. It comprises:
- Masked Self-Attention: Models dependencies between characters within the output sequence.
- 2D Attention: This is the key component that links the encoder's 2D feature map to the decoder. It allows the decoder to focus on relevant spatial regions of the image based on the current decoding step and the holistic representation.
- Point-wise Feed-Forward Layer: Standard component for processing features.
- Holistic Representation Guidance: The holistic representation is fed into the decoder at each step, guiding the 2D attention mechanism to focus on more accurate regions for character prediction.
- Training: The model is trained end-to-end using only word-level annotations, which is a significant advantage as character-level annotations are often scarce.
Datasets
The model is trained on synthetic datasets and evaluated on several standard benchmarks:
- Training:
- Synth90K (9-million-word synthetic dataset)
- SynthText (8-million-word synthetic dataset)
- Evaluation:
- Regular Text: IIIT 5K-Words (IIIT5K)
- Irregular Text: Street View Text (SVT), ICDAR2013 (IC13), ICDAR2015 (IC15), Street View Text Perspective (SVTP), CUTE80 (CT80).
- Simplified versions of IC15 (IC15-Char&Digit, IC15-1811) are also used for fair comparison.
Key Insights and Ideas
- Holistic Representation as a Contextual Guide: The idea of a single, compressed "holistic representation" of the entire text image acting as a contextual guide for the attention mechanism is a novel way to improve focus and accuracy, especially for irregular text.
- Direct 2D-to-1D Mapping: Bypassing the need to convert irregular text into a 1D sequence representation and instead directly mapping 2D image features to a 1D output sequence is more natural for irregular text and avoids potential information loss.
- Non-Recurrent Architecture for Speed: Eliminating RNNs allows for parallel training, leading to significant speedups in both forward and backward passes, making the model more practical for large-scale applications.
- Word-Level Annotations Suffice: The model's ability to achieve state-of-the-art results with only word-level annotations is a major practical advantage, as such annotations are much easier to obtain than character-level ones.
Key Results
- Performance: The proposed model achieves state-of-the-art or competitive performance on various regular and irregular scene text recognition benchmarks. For example, it shows a 3.4% accuracy increase on IC15-1811 and 1.0% on CT80 compared to other word-level approaches.
- Speed: The model is significantly faster than RNN-based counterparts, achieving 1.5x to 9.4x acceleration in the backward pass and 1.3x to 7.9x acceleration in the forward pass.
- Ablation Studies:
- ResNet34 was found to offer a good balance between model size and accuracy as the backbone.
- Using only one decoder block (N=1) achieved saturated performance, unlike the Transformer which benefits from more blocks.
- Increasing attention heads (H) generally improved performance up to 16 heads.
- The holistic representation significantly improves attention focus and accuracy, as shown in qualitative examples.
- Text rectification methods did not improve performance, suggesting the 2D attention mechanism is already adept at handling irregular text.
Limitations and Future Work
- Vertical Text: The paper notes that recognition accuracy for vertical text is very low, primarily due to a lack of sufficient vertical text samples in the current synthetic training datasets.
- Future Work: The authors suggest several avenues for future research:
- Incorporating multi-scale image features.
- Resizing input images while preserving aspect ratios.
- Further exploring the impact of self-attention in different contexts.
- Investigating the use of bidirectional decoders more thoroughly.
Summary updated