1st Place Solution to ECCV 2022 Challenge on Out of Vocabulary Scene Text Understanding: End-to-End Recognition of Out of Vocabulary Words

Zhangzi Zhu, Chuhui Xue, Yu Hao, Wenqing Zhang, Song BaiarXiv 2022 · OpenReview

Abstract

Scene text recognition has attracted increasing interest in recent years due to its wide range of applications in multilingual translation, autonomous driving, etc. In this report, we describe our solution to the Out of Vocabulary Scene Text Understanding (OOV-ST) Challenge, which aims to extract out-of-vocabulary (OOV) words from natural scene images. Our oCLIP-based model achieves 28.59% in h-mean which ranks 1st in end-to-end OOV word recognition track of OOV Challenge in ECCV2022 TiE Workshop.

Introduction: The Challenge of Out-of-Vocabulary Text

Most scene text recognition (STR) systems rely heavily on language priors. While effective for common "In-Vocabulary" (IV) words, these systems often fail on "Out-of-Vocabulary" (OOV) words—such as rare surnames, technical jargon, or random character strings—by incorrectly "correcting" them into common words. The ECCV 2022 OOV-ST Challenge was designed to push the field toward systems that recognize text based purely on visual appearance, ensuring robustness for rare or unseen words.

Proposed Methodology

The winning pipeline by the ByteDance team consists of three primary stages:

1. Vision-Language Pre-training (oCLIP)

Rather than using standard ImageNet weights, the backbones (VAN-large and Deformable ResNet-101) were pre-trained using oCLIP. This method aligns visual features with textual descriptions, providing a stronger foundation for identifying text-like structures in complex scenes.

2. Detection Ensemble

The team fine-tuned three distinct state-of-the-art detection architectures:

  • PAN: A kernel-based segmentation method.
  • Mask TextSpotter-v3 (MTS-v3): A spotter treating text as a sequence of masks.
  • TESTR: A transformer-based text spotter.

Results were merged using soft-NMS across multiple image scales (512 to 1600 pixels) to ensure high-accuracy bounding box generation.

3. SCATTER-based Recognition

For transcription, a recognizer based on the SCATTER architecture was employed. SCATTER focuses on selective spatial attention and is designed to handle irregular text shapes, making it less biased by a fixed vocabulary.

Key Insights and Results

The Impact of oCLIP Pre-training

A critical finding was that oCLIP pre-training consistently improved detection F-scores by 1% to 3% across different architectures. This demonstrates that pre-training on the relationship between vision and language helps models generalize to unseen words.

Comparison of detection performance with and without oCLIP pre-training across different base methods.

Ensemble Performance

The diversity of the ensemble (segmentation-based vs. transformer-based) proved vital. On the validation set, the ensemble reached a 72.89% F-score for OOV word detection, significantly outperforming any single model.

Performance of individual detection models versus the final ensemble on the OOV validation set.

Final Standings

The approach secured 1st Place in the end-to-end OOV word recognition track with a final 28.59% H-mean on the test set. While lower than typical OCR benchmarks, this score reflects the extreme difficulty of dictionary-free recognition.

Conclusion

The ByteDance team's success highlights the importance of vision-language pre-training and architectural diversity in tackling OOV text. While a significant gap remains between detection (72.89% F-score) and character-level recognition (28.59% H-mean), this work provides a robust framework for future "dictionary-free" scene text understanding.

Summary updated

Abstract

Scene text recognition has attracted increasing interest in recent years due to its wide range of applications in multilingual translation, autonomous driving, etc. In this report, we describe our solution to the Out of Vocabulary Scene Text Understanding (OOV-ST) Challenge, which aims to extract out-of-vocabulary (OOV) words from natural scene images. Our oCLIP-based model achieves 28.59\% in h-mean which ranks 1st in end-to-end OOV word recognition track of OOV Challenge in ECCV2022 TiE Workshop.
BibTeX
@misc{zhu2022st,
  author = {Zhangzi Zhu and Chuhui Xue and Yu Hao and Wenqing Zhang and Song Bai},
  title = {{1st Place Solution to ECCV 2022 Challenge on Out of Vocabulary Scene Text Understanding: End-to-End Recognition of Out of Vocabulary Words}},
  year = {2022},
  eprint = {2209.00224},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.00224},
}
Figure