TL;DR
Researchers have developed a new, high-performance SentencePiece Unigram tokenizer for Sindhi, a low-resource Indo-Aryan language. By benchmarking it against 14 existing tokenizers, they achieved superior efficiency (lower fertility) and discovered that several popular models were "silently" deleting characters during the tokenization process.
What Problem Does This Paper Address?
Tokenization is the foundational step in natural language processing, yet for low-resource languages like Sindhi, it remains a significant hurdle. Sindhi uses an extended Perso-Arabic script with unique characters (like implosives and specific aspirated consonants) and is morphologically rich.
The authors identify two main gaps:
- Lack of Benchmarking: While several Sindhi tokenizers exist on platforms like Hugging Face, they haven't been compared on standardized data using consistent metrics.
- Suboptimal Efficiency: General multilingual models (like XLM-RoBERTa or Llama) often over-segment low-resource text, leading to high "fertility" (too many tokens per word), which increases computational costs and shrinks the effective context window.
Methods
The authors developed a SentencePiece Unigram tokenizer with byte fallback. Unlike the common Byte-Pair Encoding (BPE) which builds a vocabulary by merging frequent pairs, the Unigram approach starts with a large vocabulary and prunes it based on a language model's likelihood.
Key technical choices included:
- Byte Fallback: Ensuring that any character not in the vocabulary is represented by its UTF-8 bytes rather than a generic "unknown" (
[UNK]) token. - Full Character Coverage: They ensured every character seen in the training data had its own piece in the model.
- Vocabulary Scaling: They trained models at four sizes: 32k, 64k (the recommended default), 100k, and 150k.
- Cross-lingual Slice: To handle loanwords and shared script features, they included a small, repeated portion of Urdu, Persian, and Arabic text in the training corpus.
Datasets
Training Data: * ~1.1 million Sindhi sentences from Wikipedia, Twitter, and news articles. * A "cross-lingual slice" of Urdu, Persian, and Arabic Wikipedia articles (making up 16.1% of the final corpus by character count).
Evaluation Benchmarks: * FLORES+: A standard translated benchmark. * Sindhi Wikipedia: Encyclopedic text. * Sentiment Set: A news-derived corpus built by the authors. * Sindh Line: A large contemporary news corpus.
Key Insights and Ideas
- The Danger of "Silent Deletion": The most striking finding was that some existing tokenizers (like TinyLlama-Sindhi and FahadQazi BPE) were deleting characters during the round-trip of encoding and decoding. For example, TinyLlama-Sindhi deleted every instance of
U+06C1(ARABIC LETTER HEH GOAL), a character common in Sindhi function words. Standard metrics often miss this because the model simply "ignores" what it can't represent. - Unigram > BPE for Morphology: While not a direct ablation, the results support previous findings that Unigram tokenization often recovers units closer to actual language morphology compared to BPE.
- Forcing Doesn't Help: The authors tried "forcing" specific frequent clitics and loanwords to be single tokens, but found that the Unigram objective naturally achieved this without manual intervention.
- Diminishing Returns on Vocab Size: Moving from a 64k to a 150k vocabulary only improved fertility by about 1.9%, suggesting that 64k is a "sweet spot" for balancing efficiency and model size.
Key Results
- Efficiency: At a 64k vocabulary size, the new tokenizer averaged 1.147 tokens per word. This is a 24.1% improvement over XLM-RoBERTa (1.512) and significantly better than the best existing Sindhi-specific baseline (1.208).
- Retention: It kept 88.6% of words as single tokens, compared to 84.4% for the nearest competitor.
- Robustness: The tokenizer achieved zero character loss across all evaluation sets, whereas some baselines lost up to 0.85% of characters.
- Outperformance: The model was the best or tied for best across almost all metrics (fertility, retention, UNK rate) in all four evaluation sets at sizes 64k and above.
Limitations and Future Work
- Intrinsic Metrics Only: The study focuses on how well the tokenizer represents text (fertility, loss), but does not test how these improvements translate to performance on downstream tasks like translation or text generation.
- No Significance Testing: While the margins appear clear, the authors did not perform formal statistical significance tests.
- Script Scope: The work is limited to the Arabic-script version of Sindhi and does not cover the Devanagari script used in India.
- Future Directions: The authors suggest training small language models using these tokenizers to validate the intrinsic gains and expanding the evaluation to include classical or literary Sindhi.
Code and tokenizer files are public: DOI: 10.5281/zenodo.23962474
Summary updated