TL;DR
This paper introduces the Symmetry-constrained Rectification Network (ScRN), a novel approach to improve scene text recognition, particularly for irregular text shapes. By leveraging geometric constraints like text center lines, character orientation, and scale, ScRN rectifies distorted text more accurately than previous methods, leading to state-of-the-art recognition performance on challenging datasets.
What Problem Does This Paper Address?
Recognizing text in natural images is a notoriously difficult task due to variations in lighting, perspective, fonts, and text shape. While many methods focus on recognizing straight or slightly oriented text, handling highly curved or distorted text instances remains a significant challenge. A common strategy is to first "rectify" these irregular text shapes into a more standard, canonical form before feeding them into a recognition model. However, existing rectification methods, often based on Spatial Transformer Networks (STNs), struggle with complex distortions because they predict control points independently, neglecting inherent geometric priors like symmetry. This can lead to inaccurate rectifications and, consequently, lower recognition accuracy for challenging text.
Methods
The core contribution is the Symmetry-constrained Rectification Network (ScRN). This network operates on shared feature maps from a backbone (ResNet-50 + FPN) and predicts geometric attributes of the text instance. Unlike prior STN-based methods, ScRN explicitly incorporates symmetry constraints by utilizing:
- Text Center Line (TCL): A flexible representation that captures the pose of straight or curved text.
- Character Orientation ($\phi$): The direction perpendicular to the text's local direction, crucial for accurate rectification, especially in non-standard cases.
- Scale ($s$): The character's size.
- Text Orientation ($\theta$): The tangential direction of the text center line.
These attributes are predicted by a lightweight convolutional network. The predicted attributes are then used to generate control points for a Thin-Plate-Spline (TPS) transformation, which warps the feature maps into a regular grid. This rectified feature map is then passed to a standard attention-based recognition module (Bi-LSTM + GRU decoder). The entire system is trained end-to-end, with both a geometry loss (for attribute prediction) and a recognition loss.
Datasets
The authors trained their model on:
- SynthText [15]: A large dataset of synthetic text images.
- Synth90k [18]: Another synthetic dataset, useful for training the recognition module when geometric annotations are unavailable.
They evaluated their method on several standard benchmarks, including:
- Regular text datasets: IIIT5K, SVT, ICDAR 2003 (IC03), ICDAR 2013 (IC13).
- Irregular text datasets: ICDAR 2015 (IC15), SVT-Perspective (SVTP), CUTE80.
Key Insights and Ideas
- Symmetry is Key: The paper strongly advocates for the importance of geometric priors, particularly symmetry, in achieving robust text rectification. Neglecting these priors leads to suboptimal results in complex scenarios.
- Local Attributes for Global Shape: By focusing on local attributes like center lines and character orientations, ScRN can accurately describe the shape of irregular text instances.
- Feature-Level Rectification: Performing rectification on feature maps rather than raw images is computationally efficient and avoids redundant processing, as the backbone features are shared between rectification and recognition.
- Character Orientation Matters: The paper highlights that the character orientation, distinct from the text orientation, is crucial for accurate rectification, especially when the text is not perfectly aligned with its local direction.
Key Results
- ScRN achieves state-of-the-art performance on several challenging scene text recognition benchmarks, particularly those with a high proportion of irregular text.
- On irregular text datasets, ScRN significantly outperforms existing methods:
- ICDAR 2015: +1.8% over baseline, +0.2% over STN_supervision.
- SVT-Perspective: +3.1% over baseline, +1.1% over STN_supervision.
- CUTE80: +3.1% over baseline, +1.0% over STN_supervision.
- The rectification module adds only a negligible computational overhead (1ms difference in inference time compared to a baseline without rectification).
- Qualitative results show ScRN producing more readable rectified images for severely distorted text compared to STN-based methods.
Limitations and Future Work
- Rectification Imperfections: The paper acknowledges that ScRN may still struggle with text instances that have messy backgrounds or rare fonts, and that rectification can be imperfect for curved text near image borders.
- Character-Level Annotations: While the authors use synthetic data for training geometric attributes, they note that obtaining character-level annotations can be labor-intensive. However, they also mention that their method can be trained with word-level annotations (like Synth90k) to improve performance.
- Future Work: The authors aim to extend their work to an end-to-end system that can handle text instances of arbitrary shapes without explicit rectification.
Summary updated