Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks

Vani Kanjirangat, Tanja Samardzic, Ljiljana Dolamic, Fabio RinaldiEMNLP 2025

TL;DR

This paper investigates how tokenization and representation biases in multilingual Large Language Models (LLMs) affect their performance on dialectal NLP tasks. The authors find that tokenization parity (TP) is a better predictor for syntactic tasks, while information parity (IP) is better for semantic tasks, and that encoder-based models generally outperform decoder-based ones.

What Problem Does This Paper Address?

Large Language Models (LLMs) have revolutionized NLP, but their application to dialects and regional language variations remains limited. Small linguistic differences, often ignored by humans, can lead to significant performance drops for LLMs, a phenomenon known as the "dialect gap." While previous work has explored factors like data size and socio-economic influences, this paper focuses on more direct, model-internal factors: the representational biases introduced by tokenization and how LLMs represent information across languages. Understanding these biases is crucial for building more equitable and effective multilingual NLP systems.

Methods

The researchers quantify representational biases using two metrics:

  • Tokenization Parity (TP): Measures how similarly a tokenizer splits sentences across different languages. A TP close to 1 means similar tokenization lengths, indicating good vocabulary coverage. Values far from 1 suggest fragmentation and potential inefficiencies.
  • Information Parity (IP): Assesses an LLM's efficiency in compressing text in a given language relative to English, drawing on information-theoretic principles. Higher IP suggests better representation.

They then correlate these bias metrics with downstream performance on three dialectal NLP tasks: Dialect Identification (DI), Topic Classification (TC), and Extractive Question Answering (EQA). The study compares encoder-only (e.g., mBERT) and decoder-only (e.g., Llama, Phi) multilingual models, controlling for script (Latin vs. non-Latin) and resource availability (high vs. low).

Datasets

The study utilizes several datasets for evaluation, primarily from the DialectBench benchmark:

  • Dialect Identification (DI): Datasets from VarDial shared tasks, including Nuanced Arabic Dialect Identification (NADI-2023), Swiss German dialect identification (GDI), Indo-Aryan Language Identification (ILI), and multi-label DSL-ML datasets.
  • Topic Classification (TC): The SIB-200 dataset, representing 200 languages.
  • Extractive Question Answering (EQA): The SDQA dataset.

For bias metric analysis, a subset of FLORES-200 (2,000 human-translated Wikipedia sentences across 200 languages) was used.

Key Insights and Ideas

  • Encoder vs. Decoder Performance: Encoder-based models consistently outperformed decoder-based LLMs across all evaluated dialectal tasks. This suggests that the architectural differences play a significant role in handling dialectal variations.
  • TP vs. IP Predictiveness: Tokenization Parity (TP) is a better predictor for tasks relying on surface-level features like syntax and morphology (e.g., EQA), while Information Parity (IP) is more indicative of performance on semantic tasks (e.g., TC).
  • Script and Resource Influence: TP is more sensitive to script type, whereas IP is influenced by both script and resource availability.
  • Surprising DI Findings: Contrary to expectations, higher TP (more fragmentation) was positively correlated with performance on Dialect Identification (DI) for better-performing models. This is hypothesized to be because dialects often differ at the surface level, and fragmentation might help models capture these distinctions. Conversely, higher IP (better compression) showed a negative correlation with DI performance, suggesting that deeper semantic similarity can make dialects harder to distinguish.
  • Tokenizer Behavior with Non-Latin Scripts: Decoder-only models, particularly those using byte-level tokenizers like Llama, exhibit significant issues with non-Latin scripts, leading to character misinterpretations and fragmentation. This can negatively impact performance on tasks involving these languages.

Key Results

  • Performance Gap: Encoder-based models (mBERT, language-specific BERTs) achieved higher F1-scores than decoder-only LLMs (Phi-3.5, Llama-3.2) on DI, TC, and EQA tasks. For instance, in DI, language-specific BERT models scored better than mBERT in most cases, except for Swiss-German.
  • Correlation Strength: TP showed a stronger correlation with surface-level tasks (DI, EQA), while IP was more strongly associated with semantic tasks (TC).
  • Non-Latin Script Issues: Decoder-only models showed considerably higher missing character proportions for non-Latin languages compared to Latin ones, with exceptions like Gemma. SILMA, an Arabic-specific model, reported a notably lower missing proportion for Arabic.
  • Llama-3.2 Tokenizer: The Llama-3.2 tokenizer exhibited significant issues with non-Latin scripts, producing misaligned tokens and potentially breaking characters into smaller byte-level components, which can degrade performance.

Limitations and Future Work

  • Model and Task Dependency: The relationships between bias metrics and performance are highly model- and task-dependent.
  • Scope of Analysis: The study focused on specific dialectal tasks. Expanding the analysis to other language-dependent tasks could provide a more comprehensive understanding.
  • Tokenizer Improvement: The authors suggest the importance of developing language-aware, adaptive tokenizers that can mitigate pre-training biases and operate flexibly across different granularities.
  • Further Investigation: Future work could explore additional use cases by identifying and incorporating relevant tasks and leveraging more extensive and diverse corpora for tokenization analysis.

Summary updated

BibTeX
@inproceedings{kanjirangat2025tokenization,
  author = {Vani Kanjirangat and Tanja Samardzic and Ljiljana Dolamic and Fabio Rinaldi},
  title = {{Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure