Robustly Recognizing Irregular Scene Text by Rectifying Principle Irregularities

Changsheng Xu, Yang Wang, Fan Bai, Jihong Guan, Shuigeng Zhou

Introduction

Scene Text Recognition (STR) has achieved high accuracy for regular, horizontal text. However, irregular text—characterized by curvature, perspective distortion, or multi-orientation—remains a significant challenge. Traditional approaches often rely on Spatial Transformer Networks (STN) using Thin Plate Splines (TPS) to "unwarp" text. However, these methods can suffer from "over-rectification," where excessive flexibility leads to character distortion or the loss of edge information.

This paper introduces Rectifying Principle Irregularities (RPI), a framework designed to perform "rough" but robust rectification. By fitting text to predefined quadratic Bézier curves, RPI preserves vital visual features, allowing modern neural recognizers to handle minor remaining irregularities.

Comparison of RPI with other rectification methods (RARE, ASTER, ScRN) on irregular text. RPI successfully rectifies and recognizes curved and perspective-distorted text where others fail.

The RPI Framework

The RPI method is composed of four integrated modules:

  1. Densities and Orientations Estimating Module (DOEM): A U-Net-style network that predicts pixel-level character density and local text orientation.
  2. Curve Fitting Module (CFM): Instead of complex optimization, RPI selects the best fit from 37 predefined quadratic Bézier curves. Selection is based on alignment with the DOEM's density and orientation maps.
  3. Grid Sampling Module (GSM): This module generates a sampling grid to flatten the image. It employs a "boundary-aware" strategy to ensure that characters near the image edges are not clipped during rectification.
  4. Sequence Recognizing Module (SRM): A CNN-LSTM-Attention encoder-decoder architecture that performs final character recognition on the rectified image.

Key Insights

A core philosophy of RPI is the "Good Enough" Principle. The authors observed that modern recognizers are inherently robust to minor rotations (up to ±20°) and scale variations. Consequently, rectification does not need to be mathematically perfect; it only needs to remove the "principal" irregularity to bring the text into a roughly horizontal alignment. By choosing from a discrete set of candidate curves rather than continuous optimization, the model avoids "hallucinated" distortions and remains computationally efficient.

Experimental Results

The RPI model was evaluated on seven major benchmarks, including regular text (IIIT5K, SVT, IC03, IC13) and irregular text (IC15, SVTP, CUTE80).

  • State-of-the-Art Performance: RPI achieved 91.7% accuracy on the CUTE80 dataset, a benchmark specifically for curved text, outperforming the previous best method (SAR) by 2.1%.
  • General Robustness: It consistently outperformed other rectification-based models like ASTER and RARE. On the IIIT5K dataset, it reached 95.1% accuracy.
  • Efficiency: The CFM module is parameter-free and uses a fixed set of curves, making the system fast and easy to implement.

Conclusion and Future Work

While RPI excels at curved text, it shows slightly lower dominance on extreme perspective distortions when trained solely on synthetic data. Future research aims to make the framework entirely end-to-end and apply the "principal irregularity" approach to full-page text spotting. By focusing on preserving information rather than achieving perfect geometric flatness, RPI provides a more robust path for recognizing text in complex real-world scenes.

Summary updated

Abstract

Reading irregular scene text is a challenging problem in scene text recognition. Rectification is a popular measure to reduce irregularities of text in images. Existing rectification methods seek to rectify text images into a strictly regular form via free parametric transformation functions. However, they always suffer from information loss or severe deformation due to their poor constraints to the transformation functions. In our investigation, we found that CNN and attention are robust to many slight irregularities. What inspires us to propose a novel and effective rectification method that mainly rectifies the principle regularities, and leaves the slight irregularities to the CNN-LSTM-attention recognizer. Our rectification method first estimates the character densities and directions of the input image in a down-sampled map then finds a best fitting curve from a small predefined Bézier curve set, and finally rectifies the input image with a transformation function corresponding to the selected curve. Transformation functions are carefully designed so that they neither lose important visual information nor cause severe deformation. Extensive experiments on seven benchmark datasets show that our method achieves the state of the art performance in most cases, especially in curved text recognition.
BibTeX
@inproceedings{xu2022robustly,
  author = {Changsheng Xu and Yang Wang and Fan Bai and Jihong Guan and Shuigeng Zhou},
  title = {{Robustly Recognizing Irregular Scene Text by Rectifying Principle Irregularities}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00140},
}
Figure