Abstract
Traditional Natural Language Processing (NLP) is constrained by fixed vocabularies, leading to Out-of-Vocabulary (OOV) errors and high data requirements for multilingual scaling. We present MTLS (Making Texts into Linguistic Symbols), a novel pre-training method that treats text as visual symbols rather than discrete token IDs. By rendering text into pixel-based patches and mapping them into the embedding space of frozen monolingual models (e.g., BERT), MTLS enables significant multilingual capabilities using only 12,000 English training sentences, bypassing the need for massive parallel corpora.
Introduction
Standard NLP models rely on a lookup-table vocabulary. This approach faces two primary hurdles: the OOV problem, where models fail on unseen scripts or words, and the multilingual scaling trap, which requires enormous vocabularies and datasets to support diverse languages.
Linguistically, language is a symbolic system. While the specific symbols (scripts) vary, the underlying semantic meanings are often universal. MTLS leverages this by treating text as visual images. This "Meta-Symbol System" allows a model to process any language by "seeing" its visual structure, effectively using the visual medium as a universal bridge between different scripts.
Methodology: The SSS Embedding
To replace the standard embedding layer of Pre-trained Language Models (PLMs), we propose the SSS Embedding framework, consisting of three core components:
- Symbolic Embedding: Text is rendered into pixel images and divided into patches. A 2D convolution operation transforms these patches into initial vector representations, eliminating the need for a fixed vocabulary.
- Selective Embedding: To account for the varying visual structures of different languages, a Mixture-of-Experts (MoE) layer is employed. A Gate Router dynamically activates specific "experts" to handle different symbolization rules, providing the necessary bias for diverse scripts.
- Spatial Embedding: This component aligns the new symbolic representations with the semantic space of a frozen PLM. It uses a two-step process: Distributional Similarity Loss ($L_{dist}$) to match the statistical distribution of original embeddings, and Spatial Similarity Loss ($L_{spat}$) via contrastive learning to ensure visual symbols (e.g., the word "love" in different scripts) map to the correct semantic coordinates.
Experimental Setup and Results
MTLS was pre-trained using a remarkably small dataset of 12,000 English sentences from the Universal Dependencies (UD) treebanks. Evaluation was conducted across 20 languages for Part-of-Speech (POS) tagging and Named Entity Recognition (NER).
Key Findings
- Script Agnosticism: MTLS showed massive gains in non-Latin scripts. For the Coptic language (COP), MTLS-BERT improved POS tagging accuracy by 66.6% over standard BERT.
- Zero-Shot Efficiency: In cross-lingual zero-shot settings, MTLS-BERT achieved an average accuracy of 24.0, significantly outperforming the base BERT's 17.2.
- Resource Efficiency: Despite the minimal training data, MTLS remained competitive with, and sometimes outperformed, massive multilingual models like mBERT and XLM-R on low-resource languages.
Discussion and Limitations
While MTLS offers a path toward data-efficient multilinguality, it introduces a "Latin script tax," where English performance slightly decreases due to the shift from optimized tokens to pixel rendering. Additionally, the computational overhead of image rendering and the current focus on encoding (understanding) rather than decoding (generation) remain areas for future optimization. By treating text as a visual symbolic system, MTLS demonstrates that models can learn to understand global languages by simply learning the visual mapping to a known semantic space.
Summary updated