Runner-Up Solution to ECCV 2022 Challenge on Out of Vocabulary Scene Text Understanding: Cropped Word Recognition

Zhangzi Zhu, Yu Hao, Wenqing Zhang, Chuhui Xue, Song BaiarXiv 2022 · OpenReview

Abstract

This report presents the 2nd place solution to the ECCV 2022 challenge on Out-of-Vocabulary Scene Text Understanding (OOV-ST): Cropped Word Recognition. The challenge, held during the Text in Everything (TiE) workshop, focuses on the difficult task of extracting out-of-vocabulary (OOV) words from natural scene images. Our approach utilizes the SCATTER architecture, pre-trained on massive synthetic datasets and fine-tuned with aggressive data augmentation. We further enhance performance by training specialized models for long and vertical text and employing a multi-model ensemble strategy. Our solution achieved a word accuracy of 59.45% on OOV words.

Introduction

Scene Text Recognition (STR) has seen significant advancements, yet it remains plagued by "vocabulary bias." Most current models excel at recognizing In-Vocabulary (IV) words—those frequently seen during training—but struggle with Out-of-Vocabulary (OOV) words like technical jargon, rare names, or random character strings. This is often because models learn to "autocorrect" unfamiliar visual inputs into known linguistic patterns. The ECCV 2022 OOV-ST challenge was designed to push the boundaries of models in recognizing these unseen words.

Methodology

Our solution is built upon the SCATTER architecture, which is known for its effectiveness in handling complex scene text through selective context attentional blocks. Our pipeline consists of four primary components:

1. Massive Synthetic Pre-training

To improve the model's ability to generalize across diverse character combinations, we generated 60 million synthetic images using a dictionary of the 90,000 most frequent English words. This large-scale pre-training ensures the model learns robust character features before fine-tuning on real-world data.

2. Aggressive Data Augmentation

Following the ABINet methodology, we applied heavy transformations during the training phase. These included geometric shifts (rotation, perspective), image quality degradation (blur, noise, JPEG compression), and color jittering. This helps the model remain invariant to the environmental noise typical of natural scene images.

3. Specialized "Expert" Models

Standard STR models often underperform on text with extreme aspect ratios. To address this, we trained two specialized versions of SCATTER:

  • Long Text Model: Optimized for images with aspect ratios greater than 9:1.
  • Vertical Text Model: Optimized for images with aspect ratios smaller than 1:3. These images were rotated 90 degrees to be processed horizontally during both training and inference.

4. Ensemble Strategy

We combined outputs from multiple models using different backbones (ResNet18 and ResNet50), input types (Grayscale and RGB), and random seeds. We also utilized "internal" ensembling by averaging the output probabilities from the five different decoder blocks within the SCATTER architecture.

Ablation study showing the incremental impact of data augmentation, synthetic data, and specialized models on validation performance.

Results and Ablation Study

Our final solution demonstrated strong performance across both IV and OOV categories. As shown in the ablation study (Figure 4), each component of our pipeline contributed to the final accuracy. Moving from a baseline SCATTER model to one incorporating data augmentation and synthetic data improved the Correctly Recognized Words (CRW) score from 54.7% to 59.3%. The addition of specialized models for long and vertical text further pushed the validation score to 60.0%.

Final performance of the proposed solution on IV and OOV datasets.

In the final competition evaluation, our solution achieved an OOV word accuracy of 59.45% and an overall accuracy (IV & OOV) of 69.6%. The results highlight that while OOV recognition remains more challenging than IV recognition (79.7% accuracy), the combination of massive synthetic data and specialized architectural handling significantly closes the gap.

Conclusion

The ByteDance team's runner-up solution demonstrates that overcoming vocabulary bias in STR requires a multi-faceted approach. By combining robust architectural foundations (SCATTER) with massive synthetic pre-training and specialized handling for extreme aspect ratios, we can significantly improve the recognition of previously unseen words. Future work will likely involve tighter integration between text detection and recognition to further enhance end-to-end performance in complex scenes.

Summary updated

Abstract

This report presents our 2nd place solution to ECCV 2022 challenge on Out-of-Vocabulary Scene Text Understanding (OOV-ST) : Cropped Word Recognition. This challenge is held in the context of ECCV 2022 workshop on Text in Everything (TiE), which aims to extract out-of-vocabulary words from natural scene images. In the competition, we first pre-train SCATTER on the synthetic datasets, then fine-tune the model on the training set with data augmentations. Meanwhile, two additional models are trained specifically for long and vertical texts. Finally, we combine the output from different models with different layers, different backbones, and different seeds as the final results. Our solution achieves a word accuracy of 59.45\% when considering out-of-vocabulary words only.
BibTeX
@misc{zhu2022runner,
  author = {Zhangzi Zhu and Yu Hao and Wenqing Zhang and Chuhui Xue and Song Bai},
  title = {{Runner-Up Solution to ECCV 2022 Challenge on Out of Vocabulary Scene Text Understanding: Cropped Word Recognition}},
  year = {2022},
  eprint = {2208.02747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.02747},
}
Figure