Abstract
MENTOR is a novel multilingual text detection framework designed to identify text in languages unseen during training without requiring human-labeled data or model re-training. By utilizing "zero-cost" synthesized printed text as a reference, the model learns to identify matching patterns in complex scene images through a "learning by analogy" approach. This framework effectively brings zero-shot learning to the domain of Optical Character Recognition (OCR), addressing the scalability issues inherent in traditional supervised methods.
Introduction: The Challenge of Multilingual Text Detection
As autonomous systems and delivery vehicles navigate increasingly globalized urban environments, they encounter a diverse array of scripts and languages. Standard text detection models are typically trained on a fixed set of languages, often with a heavy bias toward English. When these systems encounter a new script—such as Malayalam or Greek—the traditional solution involves collecting thousands of human-annotated images and re-training the model. This process is not only expensive and slow but also prone to "catastrophic forgetting," where the model loses proficiency in previously learned languages.
MENTOR addresses this by asking: Can we build a detector that generalizes to unseen languages by simply showing it what that language looks like in a standard font? This approach mimics human behavior; a person can often locate characters of an unfamiliar script on a sign if they are first shown a reference sheet of those characters.
The MENTOR Architecture: Learning by Analogy
The core philosophy of MENTOR is to enable detection through visual analogy. The architecture consists of three primary components that work in tandem to reconfigure the detector for new languages at inference time.
1. Dynamic Guide (DG)
The Dynamic Guide processes "zero-cost" images—text rendered from standard computer fonts—of the target language. Using a ResNet34 backbone, it extracts a language-specific representation ($Match_{print}$) that captures the essential visual style and geometric properties of the script.
2. Text Finder (TF)
The Text Finder employs a two-pronged approach. One branch ($Sub^{scene}_{detect}$) is language-agnostic, identifying general "text-like" regions in a scene image using a ResNet50+FPN backbone. The second branch ($Sub^{scene}_{language}$) extracts deep features from the scene image to be matched against the printed reference.
3. Language Mapper (LM)
The Language Mapper acts as the system's central processor. It transforms the representation from the Dynamic Guide into dynamic convolution kernels. These kernels serve as specialized filters applied to the scene features ($Match_{scene}$). If the scene features align with the analogy provided by the printed text, the model flags the region as a detection. This dynamic reconfigurability allows the model to adapt to new languages simply by swapping the input printed text images, without modifying the underlying weights.
Experimental Evaluation
The researchers evaluated MENTOR across several benchmarks to test its generalization capabilities:
- ICDAR 2017 & 2019 MLT: These large-scale multilingual datasets were used to train the model on "seen" languages and evaluate its performance on "unseen" ones.
- IIIT-ILST: This dataset was used to test the model on Malayalam, a language entirely absent from the training set, providing a rigorous test for zero-shot capability.
- Synthetic MLT17: A custom-augmented dataset designed to evaluate the model's ability to distinguish between multiple languages appearing within the same frame.
Key Results
MENTOR demonstrated significant improvements over existing methods in zero-shot scenarios:
- Unseen Language Performance: When tested on Korean as an unseen language, MENTOR achieved an F-score of 72.48%, outperforming the state-of-the-art MultiplexerOCR (67.86%).
- Zero-Shot Success: On the Malayalam dataset, MENTOR reached a 65.84% F-score. In contrast, traditional supervised methods that had not seen Malayalam performed poorly, with some scores as low as 11.9%.
- Competitive Performance on Seen Languages: MENTOR remains competitive with fully supervised models for languages included in its training set (e.g., English, Chinese), proving that the analogy-based approach does not compromise base performance.
Discussion and Future Work
One of the most elegant insights of MENTOR is the use of printed text as a "free" mentor. Digital fonts contain the visual "DNA" of every language, and leveraging them bypasses the need for manual scene-text labeling. The researchers also found that training with "intersection" synthetic data—pasting synthetic text of the same language into a scene—was crucial for forcing the model to learn intrinsic character geometry.
However, limitations remain. At 3.25 FPS, MENTOR is currently too slow for real-time high-speed robotics. The multi-stage matching process introduces significant computational overhead. Additionally, the model can struggle to distinguish between very similar-looking scripts (e.g., CJK scripts) in low-resolution or cluttered scenes. Future research will focus on more efficient backbone architectures and extending the analogy framework to full text recognition.
Summary updated