This paper introduces Saidzoda-Tajik-E5-Large, the first openly described text embedding model specifically optimized for the Tajik language. By fine-tuning a 560M parameter multilingual model using synthetically generated data, the authors demonstrate that a relatively small, efficient model can match or outperform models up to 14 times its size on specialized Tajik retrieval tasks.
TL;DR
Researchers have developed a specialized Tajik text embedder by fine-tuning multilingual-e5-large-instruct on over 235,000 synthetic question-passage pairs. The resulting model significantly improves retrieval accuracy for Tajik laws and tax documents while remaining efficient enough to run on modest hardware.
What Problem Does This Paper Address?
Tajik, a language spoken by millions and written in a unique Cyrillic script, is often an "afterthought" in the world of Large Language Models (LLMs). The authors identify two major hurdles:
- Tokenization Gaps: Popular multilingual tokenizers (like those for E5 or BGE-M3) fail to recognize 4 out of the 12 unique Tajik letters (ҷ, ӣ, ӯ, ҷ), often collapsing them into a single "unknown" token.
- Lack of Benchmarks: There were no existing public benchmarks or models specifically designed for Tajik retrieval tasks, such as finding specific legal articles or tax regulations.
Methods
The authors started with the multilingual-e5-large-instruct (560M) model and applied a "recipe" for domain adaptation:
- Synthetic Query Generation: Since human-labeled Tajik retrieval data is scarce, they used a Tajik-tuned LLM (Qwen3.6-27B) to generate questions for passages from Tajik laws, Wikipedia, and web text.
- Contrastive Training: The model was trained using a multiple-negatives ranking loss. To make the task harder and the model sharper, they mined "hard negatives"—passages that look relevant but are actually incorrect—using an existing embedder.
- Language Preservation (Replay): To ensure the model didn't "forget" how to handle English and Russian, they mixed in existing datasets from those languages (NQ and MIRACL) during training.
- Precision Matters: Interestingly, the authors found that training in
bfloat16caused the model to collapse (loss dropped to zero), forcing them to usefp32for stable training.
Datasets
The study utilized several Tajik-specific collections:
- Tajik Law: 72,920 chunks of legal text.
- Tajik Tax: 23,282 chunks of tax codes and guidance.
- Tajik Wikipedia & Web Text: General knowledge and literature.
- Belebele: A human-written reading comprehension dataset used for independent evaluation.
- Replay Sets: English Natural Questions (NQ) and Russian MIRACL to maintain multilingual performance.
Key Insights and Ideas
- The "Unknown" Token Tax: General multilingual models are effectively "blind" to certain Tajik nuances because their tokenizers weren't built with the language in mind.
- Synthetic Data as a Bridge: A high-quality LLM in the target language can effectively "bootstrap" a retrieval model by creating massive amounts of training data where none existed.
- Efficiency vs. Scale: You don't always need a multi-billion parameter model. A 560M model, when properly exposed to a low-resource language, can go toe-to-toe with 8B parameter giants.
- Folding Fails: A common trick is "folding" unique Tajik letters into similar-looking Russian ones (e.g., mapping ҷ to ч). The authors tested this and found it actually decreased performance, suggesting the model benefits from learning the distinct Tajik characters.
Key Results
The fine-tuned model showed dramatic improvements over the base E5 model across several Tajik-specific test sets:
- Law Retrieval: Recall@10 jumped from 80.2% to 93.6%.
- Tax Retrieval: Recall@10 rose from 82.6% to 94.5%.
- Human-Written Test (Belebele): Performance improved from 92.2% to 95.2%.
- Efficiency: The model processes queries in ~12ms on an H100, roughly 3x faster than the 3B and 8B parameter competitors.
As shown in Table 3, the tuned 560M model (Ours) is competitive with much larger models like Giga-Embeddings-3B and Qwen3-Embedding-8B on Tajik tasks.
Limitations and Future Work
- Synthetic Bias: The law, tax, and Wikipedia test questions were generated by the same LLM used for training. This likely provides an "upper bound" on performance; the model might be learning the LLM's "style" rather than just the language.
- Need for Human Data: The most critical next step is developing a retrieval benchmark where the questions are written by native Tajik speakers to validate these gains in real-world scenarios.
- Scope: The study focused on retrieval only, not the subsequent "re-ranking" or "generation" steps of a full RAG (Retrieval-Augmented Generation) pipeline.
Summary updated