Classifying the Unknown: In-Context Learning for Open-Vocabulary Text and Symbol Recognition

Tom Simon, William Mocaër, Pierrick Tranouez, Clément Chatelain, Thierry Paquet

Abstract

Rosetta is a multimodal model designed for open-vocabulary symbol recognition using In-Context Learning (ICL). By employing a novel Context-Aware Tokenizer (CAT), Rosetta maps query symbols to positions in a reference image, enabling zero-shot adaptation to unseen scripts and fonts without retraining. This approach shifts the recognition paradigm from fixed-vocabulary memorization to visual analogy, significantly outperforming traditional OCR systems on out-of-distribution data.

Introduction: The "Stiffness" of Traditional OCR

Traditional Optical Character Recognition (OCR) systems are inherently rigid. They are typically trained on fixed vocabularies and specific scripts. When these models encounter rare scripts, decorative fonts, or abstract symbols not present in their training data, performance degrades significantly. Addressing this usually requires costly data collection and fine-tuning.

Rosetta introduces a paradigm shift from memorization to analogy. Inspired by the Rosetta Stone, which allowed scholars to decode hieroglyphs by comparing them to known Greek text, this model uses In-Context Learning to recognize unseen symbols by referencing a provided "key" image. This allows the model to "read" entirely new alphabets simply by looking at a reference string.

The Rosetta framework utilizes a context image (Xc, Yc) as a reference to decode a query image (X) into the target text (Y).

Methodology: The Context-Driven Paradigm

The core innovation of Rosetta is the Context-Driven Classification Paradigm. Instead of predicting a character class directly (e.g., mapping a shape to the letter 'A'), the model predicts a token representing a position in the provided context image.

Context-Aware Tokenizer (CAT)

The "secret sauce" of Rosetta is the Context-Aware Tokenizer (CAT). Unlike standard tokenizers with static vocabularies, CAT dynamically maps characters to tokens based on their order of appearance in a reference string. As illustrated in the iterative decoding process, characters in the context $Y_c$ are assigned tokens $\langle t_i \rangle$. The model then predicts these tokens for the query image $X$. If a symbol in the query does not appear in the context, it is labeled as <ooc> (out-of-context).

The Context-Aware Tokenizer (CAT) maps characters to position-based tokens, allowing the model to decode symbols by matching them to their positions in the reference context.

Visual Prompt Generator (VPG) and Decoder

Rosetta utilizes the Qwen2-VL architecture as its backbone. A Visual Prompt Generator (VPG) converts both context and query images into visual tokens, employing 2D Rotary Positional Embeddings (2D-RoPE) to preserve spatial relationships. These tokens are then processed by a 240-million parameter Transformer Decoder to predict the sequence of symbols based on the provided context.

Experimental Results

To ensure the model learned visual matching rather than linguistic patterns, it was trained on synthetic datasets of random character sequences across 4,000 distinct fonts. This forced the model to rely purely on visual analogy.

Performance on Unseen Data

Rosetta demonstrated remarkable generalization capabilities across several evaluation sets:

  • Unseen Fonts: On new Latin fonts, Rosetta achieved a Character Error Rate (CER) of 6.54%, nearly halving the 13.56% error rate of a standard fixed-vocabulary baseline.
  • Zero-Shot Script Adaptation: Despite being trained primarily on Latin characters, the model successfully classified Japanese, Chinese, Greek, and Russian scripts when provided with the appropriate context.
  • Robustness: The model maintained a Token Error Rate (TER) below 10% even when faced with abstract, non-alphabetic symbols, proving its effectiveness as a general-purpose pattern matcher.

Key Insights and Limitations

The primary insight of this work is that learning to match is more powerful than learning to read for open-vocabulary tasks. By focusing on the relationship between context and query, Rosetta becomes an "open-vocabulary" system by design, theoretically capable of handling an infinite number of classes as long as they are defined in the context.

However, several limitations remain. The current model was evaluated on clean, synthetic data; real-world noise like blur, low resolution, or physical damage presents a further challenge. Additionally, the system currently relies on high-quality, manually provided context images. Future work will likely focus on automated context selection and scaling the architecture for large-scale industrial applications.

Summary updated

BibTeX
@inproceedings{simon2025classifying,
  author = {Tom Simon and William Mocaër and Pierrick Tranouez and Clément Chatelain and Thierry Paquet},
  title = {{Classifying the Unknown: In-Context Learning for Open-Vocabulary Text and Symbol Recognition}},
  booktitle = {ICDAR},
  year = {2025},
  doi = {10.1007/978-3-032-04627-7\_13},
}

Loading related content...

Figure