I2C2W: Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition

Chuhui Xue, Shijian Lu, Song Bai, Wenqing Zhang, Changhu Wang

Abstract Leveraging the advances of natural language processing, most recent scene text recognizers adopt an encoder-decoder architecture where text images are first converted to representative features and then a sequence of characters via 'sequential decoding'. However, scene text images suffer from rich noises of different sources such as complex background and geometric distortions which often confuse the decoder and lead to incorrect alignment of visual features at noisy decoding time steps. This paper presents I2C2W, a novel scene text recognition technique that is tolerant to geometric and photometric degradation by decomposing scene text recognition into two inter-connected tasks. The first task focuses on image-to-character (I2C) mapping which detects a set of character candidates from images based on different alignments of visual features in an non-sequential way. The second task tackles character-to-word (C2W) mapping which recognizes scene text by decoding words from the detected character candidates. The direct learning from character semantics (instead of noisy image features) corrects falsely detected character candidates effectively which improves the final text recognition accuracy greatly. Extensive experiments over nine public datasets show that the proposed I2C2W outperforms the state-of-the-art by large margins for challenging scene text datasets with various curvature and perspective distortions. It also achieves very competitive recognition performance over multiple normal scene text datasets.

Introduction

Scene text recognition (STR) has traditionally relied on sequential decoding, where characters are predicted one by one. This approach depends heavily on the model's ability to maintain precise visual attention at each step. However, real-world images often contain complex backgrounds, glares, and significant geometric distortions—such as text on curved surfaces or at extreme angles. These factors frequently lead to "misalignment" in sequential models, where the decoder loses its place, causing cascading errors.

I2C2W addresses these challenges by abandoning the strict sequential constraint. Instead, it decomposes recognition into two stages: non-sequential character detection followed by semantic refinement. This "spell-checking" mechanism allows the model to correct visual misinterpretations using learned linguistic patterns, making it exceptionally robust to irregular text.

The I2C2W Architecture

The framework consists of two primary Transformer-based modules: the Image-to-Character (I2C) module and the Character-to-Word (C2W) module.

The I2C2W architecture consists of an Image-to-Character (I2C) module for non-sequential detection and a Character-to-Word (C2W) module for semantic refinement.
  1. Image-to-Character (I2C) Module: This module treats character detection as a parallel set-prediction task rather than a sequence. Utilizing a ResNet-50 backbone and a Transformer encoder-decoder, it identifies a set of character candidates. A key innovation is that it predicts both the character category and its relative position (e.g., 1st, 2nd, or 3rd in a word) simultaneously. This process is supervised using only word-level transcripts, eliminating the need for costly character-level bounding box annotations.
  2. Character-to-Word (C2W) Module: The C2W module takes the potentially noisy candidates from the I2C stage and treats them as a sequence of embeddings. A second Transformer decoder learns the semantic relationships between these characters. For instance, if the I2C module visually misidentifies an 'a' in the word "meeting," the C2W module uses linguistic context to correct it to an 'e' before the final Connectionist Temporal Classification (CTC) output.

Experimental Results

The model was evaluated across 11 public datasets, including synthetic data for pre-training and various real-world categories: curved (CUTE80, Total-Text, SCUT-CTW1500), multi-oriented (ICDAR-2015, SVT, SVTP), and normal (IIIT5K, IC03, IC13).

Comparison of I2C2W against state-of-the-art methods on curved text datasets.

I2C2W achieved state-of-the-art (SOTA) performance on curved datasets by significant margins. On the Total-Text dataset, it reached 81.5% accuracy, a substantial improvement over previous benchmarks. Similarly, on the CUTE dataset, it achieved 93.1%, outperforming previous leaders like RobustScanner. Ablation studies confirmed that the C2W module alone improved raw I2C accuracy by 4% to 12%, demonstrating the critical role of semantic correction.

Conclusion

By breaking the traditional sequential decoding paradigm, I2C2W offers a more flexible and robust approach to scene text recognition. Reformulating character localization as a classification problem and leveraging character semantics as a "safety net" allows the model to overcome severe visual distortions. While it remains challenged by heavy occlusions or ultra-low resolution, its performance on irregular text sets a new benchmark for the field. Future work may involve integrating pre-trained language models like BERT to further enhance the C2W module's reasoning capabilities.

Summary updated

Abstract

Leveraging the advances of natural language processing, most recent scene text recognizers adopt an encoder-decoder architecture where text images are first converted to representative features and then a sequence of characters via `sequential decoding'. However, scene text images suffer from rich noises of different sources such as complex background and geometric distortions which often confuse the decoder and lead to incorrect alignment of visual features at noisy decoding time steps. This paper presents I2C2W, a novel scene text recognition technique that is tolerant to geometric and photometric degradation by decomposing scene text recognition into two inter-connected tasks. The first task focuses on image-to-character (I2C) mapping which detects a set of character candidates from images based on different alignments of visual features in an non-sequential way. The second task tackles character-to-word (C2W) mapping which recognizes scene text by decoding words from the detected character candidates. The direct learning from character semantics (instead of noisy image features) corrects falsely detected character candidates effectively which improves the final text recognition accuracy greatly. Extensive experiments over nine public datasets show that the proposed I2C2W outperforms the state-of-the-art by large margins for challenging scene text datasets with various curvature and perspective distortions. It also achieves very competitive recognition performance over multiple normal scene text datasets.
BibTeX
@article{xue2023i,
  author = {Chuhui Xue and Shijian Lu and Song Bai and Wenqing Zhang and Changhu Wang},
  title = {{I2C2W: Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3230962},
}
Figure