Abstract
Rosetta is a multimodal model designed for open-vocabulary symbol recognition using In-Context Learning (ICL). By employing a novel Context-Aware Tokenizer (CAT), Rosetta maps query symbols to positions in a reference image, enabling zero-shot adaptation to unseen scripts and fonts without retraining. This approach shifts the recognition paradigm from fixed-vocabulary memorization to visual analogy, significantly outperforming traditional OCR systems on out-of-distribution data.
Introduction: The "Stiffness" of Traditional OCR
Traditional Optical Character Recognition (OCR) systems are inherently rigid. They are typically trained on fixed vocabularies and specific scripts. When these models encounter rare scripts, decorative fonts, or abstract symbols not present in their training data, performance degrades significantly. Addressing this usually requires costly data collection and fine-tuning.
Rosetta introduces a paradigm shift from memorization to analogy. Inspired by the Rosetta Stone, which allowed scholars to decode hieroglyphs by comparing them to known Greek text, this model uses In-Context Learning to recognize unseen symbols by referencing a provided "key" image. This allows the model to "read" entirely new alphabets simply by looking at a reference string.
Methodology: The Context-Driven Paradigm
The core innovation of Rosetta is the Context-Driven Classification Paradigm. Instead of predicting a character class directly (e.g., mapping a shape to the letter 'A'), the model predicts a token representing a position in the provided context image.
Context-Aware Tokenizer (CAT)
The "secret sauce" of Rosetta is the Context-Aware Tokenizer (CAT). Unlike standard tokenizers with static vocabularies, CAT dynamically maps characters to tokens based on their order of appearance in a reference string. As illustrated in the iterative decoding process, characters in the context $Y_c$ are assigned tokens $\langle t_i \rangle$. The model then predicts these tokens for the query image $X$. If a symbol in the query does not appear in the context, it is labeled as <ooc> (out-of-context).
Visual Prompt Generator (VPG) and Decoder
Rosetta utilizes the Qwen2-VL architecture as its backbone. A Visual Prompt Generator (VPG) converts both context and query images into visual tokens, employing 2D Rotary Positional Embeddings (2D-RoPE) to preserve spatial relationships. These tokens are then processed by a 240-million parameter Transformer Decoder to predict the sequence of symbols based on the provided context.
Experimental Results
To ensure the model learned visual matching rather than linguistic patterns, it was trained on synthetic datasets of random character sequences across 4,000 distinct fonts. This forced the model to rely purely on visual analogy.
Performance on Unseen Data
Rosetta demonstrated remarkable generalization capabilities across several evaluation sets:
- Unseen Fonts: On new Latin fonts, Rosetta achieved a Character Error Rate (CER) of 6.54%, nearly halving the 13.56% error rate of a standard fixed-vocabulary baseline.
- Zero-Shot Script Adaptation: Despite being trained primarily on Latin characters, the model successfully classified Japanese, Chinese, Greek, and Russian scripts when provided with the appropriate context.
- Robustness: The model maintained a Token Error Rate (TER) below 10% even when faced with abstract, non-alphabetic symbols, proving its effectiveness as a general-purpose pattern matcher.
Key Insights and Limitations
The primary insight of this work is that learning to match is more powerful than learning to read for open-vocabulary tasks. By focusing on the relationship between context and query, Rosetta becomes an "open-vocabulary" system by design, theoretically capable of handling an infinite number of classes as long as they are defined in the context.
However, several limitations remain. The current model was evaluated on clean, synthetic data; real-world noise like blur, low resolution, or physical damage presents a further challenge. Additionally, the system currently relies on high-quality, manually provided context images. Future work will likely focus on automated context selection and scaling the architecture for large-scale industrial applications.
Summary updated