Abstract
Cross-lingual transfer is essential for extending Natural Language Processing (NLP) capabilities to low-resource languages. However, the common practice of using English as the default source language is often suboptimal. This paper introduces Subword Evenness (SuE), a novel metric that measures the uniformity of subword tokenization. Our findings indicate that languages with high subword regularity—particularly those using non-Latin scripts—are significantly more effective "universal donors" for cross-lingual transfer than English.
Introduction
In multilingual NLP, cross-lingual transfer involves fine-tuning a large model (e.g., mBERT, XLM-R) on a high-resource "source" language before applying it to a low-resource "target." While English is the traditional default source, recent evidence suggests it is a poor choice for many non-European targets. Traditional predictors of transfer success, such as genealogical or typological similarity, often fail to capture the underlying structural reasons why certain language pairs succeed. This study proposes a data-driven explanation centered on how tokenizers interact with different languages.
Methods: Defining Subword Evenness (SuE)
The authors hypothesize that the regularity of subword splits influences a model's ability to generalize across languages. They propose the Subword Evenness (SuE) metric, calculated through a four-step process:
- Tokenization: Words are decomposed into subwords using unsupervised algorithms like Byte-Pair Encoding.
- Unevenness Index (UI): For each word, the UI is defined as the difference between the lengths of the longest and shortest subword segments.
- Geometric Analysis: UI values are plotted against total word length. Kernel Density Estimation (KDE) is then used to map the distribution of these points.
- Angle Measurement: By fitting lines to the distribution's edges, an "upper angle" is measured. A low SuE score represents high regularity (even splits), while a high score indicates irregular, uneven splits.
The metric was validated using the TeDDi corpus, testing 19 high-resource transfer languages against 30 diverse low-resource target languages across three major models: mBERT, XLM-R, and mT5.
Key Results and Insights
The study reveals several critical insights into the mechanics of cross-lingual transfer:
The "Script" Factor and Evenness
A major finding is that the most successful transfer languages—such as Japanese, Mandarin, Korean, Hebrew, and Greek—all utilize non-Latin scripts. These scripts appear to impose structural constraints that force tokenizers to learn more regular, useful patterns. SuE outperformed traditional metrics (e.g., type-token ratios, entropy) in predicting transfer success.
Performance and Correlation
SuE showed a strong correlation with perplexity reduction in target languages. For the XLM-R model, the Pearson correlation reached 0.56. Transferring from low-SuE languages consistently resulted in lower average perplexity across the 30 target languages compared to transferring from English.
The English Myth
Notably, English was never among the top five transfer languages for the diverse target set. While effective for transferring to other European languages, English proved to be a mediocre choice for linguistically diverse low-resource targets. The authors suggest that training on low-SuE languages biases models toward regular structures, making them more robust to the "noise" of unfamiliar target languages.
Conclusion
Subword Evenness provides a powerful, data-driven metric for selecting source languages in cross-lingual transfer. By prioritizing languages with regular subword structures, researchers can significantly improve model performance on low-resource targets. Future work will explore whether SuE can optimize tokenization processes or improve pivot language selection in machine translation.
Summary updated