Abstract
Scene text recognition has attracted increasing interest in recent years due to its wide range of applications in multilingual translation, autonomous driving, etc. In this report, we describe our solution to the Out of Vocabulary Scene Text Understanding (OOV-ST) Challenge, which aims to extract out-of-vocabulary (OOV) words from natural scene images. Our oCLIP-based model achieves 28.59% in h-mean which ranks 1st in end-to-end OOV word recognition track of OOV Challenge in ECCV2022 TiE Workshop.
Introduction: The Challenge of Out-of-Vocabulary Text
Most scene text recognition (STR) systems rely heavily on language priors. While effective for common "In-Vocabulary" (IV) words, these systems often fail on "Out-of-Vocabulary" (OOV) words—such as rare surnames, technical jargon, or random character strings—by incorrectly "correcting" them into common words. The ECCV 2022 OOV-ST Challenge was designed to push the field toward systems that recognize text based purely on visual appearance, ensuring robustness for rare or unseen words.
Proposed Methodology
The winning pipeline by the ByteDance team consists of three primary stages:
1. Vision-Language Pre-training (oCLIP)
Rather than using standard ImageNet weights, the backbones (VAN-large and Deformable ResNet-101) were pre-trained using oCLIP. This method aligns visual features with textual descriptions, providing a stronger foundation for identifying text-like structures in complex scenes.
2. Detection Ensemble
The team fine-tuned three distinct state-of-the-art detection architectures:
- PAN: A kernel-based segmentation method.
- Mask TextSpotter-v3 (MTS-v3): A spotter treating text as a sequence of masks.
- TESTR: A transformer-based text spotter.
Results were merged using soft-NMS across multiple image scales (512 to 1600 pixels) to ensure high-accuracy bounding box generation.
3. SCATTER-based Recognition
For transcription, a recognizer based on the SCATTER architecture was employed. SCATTER focuses on selective spatial attention and is designed to handle irregular text shapes, making it less biased by a fixed vocabulary.
Key Insights and Results
The Impact of oCLIP Pre-training
A critical finding was that oCLIP pre-training consistently improved detection F-scores by 1% to 3% across different architectures. This demonstrates that pre-training on the relationship between vision and language helps models generalize to unseen words.
Ensemble Performance
The diversity of the ensemble (segmentation-based vs. transformer-based) proved vital. On the validation set, the ensemble reached a 72.89% F-score for OOV word detection, significantly outperforming any single model.
Final Standings
The approach secured 1st Place in the end-to-end OOV word recognition track with a final 28.59% H-mean on the test set. While lower than typical OCR benchmarks, this score reflects the extreme difficulty of dictionary-free recognition.
Conclusion
The ByteDance team's success highlights the importance of vision-language pre-training and architectural diversity in tackling OOV text. While a significant gap remains between detection (72.89% F-score) and character-level recognition (28.59% H-mean), this work provides a robust framework for future "dictionary-free" scene text understanding.
Summary updated