MENTOR: Multilingual tExt detectioN TOward leaRning by analogy

Hsin-Ju Lin, Tsu-Chun Chung, Ching-Chun Hsiao, Pin-Yu Chen, Wei-Chen Chiu, Ching-Chun Huang

Abstract

MENTOR is a novel multilingual text detection framework designed to identify text in languages unseen during training without requiring human-labeled data or model re-training. By utilizing "zero-cost" synthesized printed text as a reference, the model learns to identify matching patterns in complex scene images through a "learning by analogy" approach. This framework effectively brings zero-shot learning to the domain of Optical Character Recognition (OCR), addressing the scalability issues inherent in traditional supervised methods.

Introduction: The Challenge of Multilingual Text Detection

As autonomous systems and delivery vehicles navigate increasingly globalized urban environments, they encounter a diverse array of scripts and languages. Standard text detection models are typically trained on a fixed set of languages, often with a heavy bias toward English. When these systems encounter a new script—such as Malayalam or Greek—the traditional solution involves collecting thousands of human-annotated images and re-training the model. This process is not only expensive and slow but also prone to "catastrophic forgetting," where the model loses proficiency in previously learned languages.

Comparison of MENTOR with previous methods, highlighting the use of printed text as a "free" mentor for unseen languages.

MENTOR addresses this by asking: Can we build a detector that generalizes to unseen languages by simply showing it what that language looks like in a standard font? This approach mimics human behavior; a person can often locate characters of an unfamiliar script on a sign if they are first shown a reference sheet of those characters.

The MENTOR Architecture: Learning by Analogy

The core philosophy of MENTOR is to enable detection through visual analogy. The architecture consists of three primary components that work in tandem to reconfigure the detector for new languages at inference time.

The MENTOR architecture overview, featuring the Dynamic Guide, Text Finder, and Language Mapper modules.

1. Dynamic Guide (DG)

The Dynamic Guide processes "zero-cost" images—text rendered from standard computer fonts—of the target language. Using a ResNet34 backbone, it extracts a language-specific representation ($Match_{print}$) that captures the essential visual style and geometric properties of the script.

2. Text Finder (TF)

The Text Finder employs a two-pronged approach. One branch ($Sub^{scene}_{detect}$) is language-agnostic, identifying general "text-like" regions in a scene image using a ResNet50+FPN backbone. The second branch ($Sub^{scene}_{language}$) extracts deep features from the scene image to be matched against the printed reference.

3. Language Mapper (LM)

The Language Mapper acts as the system's central processor. It transforms the representation from the Dynamic Guide into dynamic convolution kernels. These kernels serve as specialized filters applied to the scene features ($Match_{scene}$). If the scene features align with the analogy provided by the printed text, the model flags the region as a detection. This dynamic reconfigurability allows the model to adapt to new languages simply by swapping the input printed text images, without modifying the underlying weights.

Experimental Evaluation

The researchers evaluated MENTOR across several benchmarks to test its generalization capabilities:

  • ICDAR 2017 & 2019 MLT: These large-scale multilingual datasets were used to train the model on "seen" languages and evaluate its performance on "unseen" ones.
  • IIIT-ILST: This dataset was used to test the model on Malayalam, a language entirely absent from the training set, providing a rigorous test for zero-shot capability.
  • Synthetic MLT17: A custom-augmented dataset designed to evaluate the model's ability to distinguish between multiple languages appearing within the same frame.

Key Results

MENTOR demonstrated significant improvements over existing methods in zero-shot scenarios:

  • Unseen Language Performance: When tested on Korean as an unseen language, MENTOR achieved an F-score of 72.48%, outperforming the state-of-the-art MultiplexerOCR (67.86%).
  • Zero-Shot Success: On the Malayalam dataset, MENTOR reached a 65.84% F-score. In contrast, traditional supervised methods that had not seen Malayalam performed poorly, with some scores as low as 11.9%.
  • Competitive Performance on Seen Languages: MENTOR remains competitive with fully supervised models for languages included in its training set (e.g., English, Chinese), proving that the analogy-based approach does not compromise base performance.

Discussion and Future Work

One of the most elegant insights of MENTOR is the use of printed text as a "free" mentor. Digital fonts contain the visual "DNA" of every language, and leveraging them bypasses the need for manual scene-text labeling. The researchers also found that training with "intersection" synthetic data—pasting synthetic text of the same language into a scene—was crucial for forcing the model to learn intrinsic character geometry.

However, limitations remain. At 3.25 FPS, MENTOR is currently too slow for real-time high-speed robotics. The multi-stage matching process introduces significant computational overhead. Additionally, the model can struggle to distinguish between very similar-looking scripts (e.g., CJK scripts) in low-resolution or cluttered scenes. Future research will focus on more efficient backbone architectures and extending the analogy framework to full text recognition.

Summary updated

Abstract

Text detection is frequently used in vision-based mobile robots when they need to interpret texts in their surroundings to perform a given task. For instance, delivery robots in multilingual cities need to be capable of doing multilingual text detection so that the robots can read traffic signs and road markings. Moreover, the target languages change from region to region, implying the need of efficiently re-training the models to recognize the novel/new languages. However, collecting and labeling training data for novel languages are cumbersome, and the efforts to re-train an existing/trained text detector are considerable. Even worse, such a routine would repeat whenever a novel language appears. This motivates us to propose a new problem setting for tackling the aforementioned challenges in a more efficient way: "We ask for a generalizable multilingual text detection framework to detect and identify both seen and unseen language regions inside scene images without the requirement of collecting supervised training data for unseen languages as well as model re-training". To this end, we propose "MENTOR", the first work to realize a learning strategy between zero-shot learning and few-shot learning for multilingual scene text detection.
BibTeX
@inproceedings{lin2023mentor,
  author = {Hsin-Ju Lin and Tsu-Chun Chung and Ching-Chun Hsiao and Pin-Yu Chen and Wei-Chen Chiu and Ching-Chun Huang},
  title = {{MENTOR: Multilingual tExt detectioN TOward leaRning by analogy}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342419},
}
Figure