Saidzoda-Tajik-E5-Large: Fine-Tuning a 560M Embedder for Tajik Retrieval

Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon ShodyevaarXiv 2026 · arXiv

This paper introduces Saidzoda-Tajik-E5-Large, the first openly described text embedding model specifically optimized for the Tajik language. By fine-tuning a 560M parameter multilingual model using synthetically generated data, the authors demonstrate that a relatively small, efficient model can match or outperform models up to 14 times its size on specialized Tajik retrieval tasks.

TL;DR

Researchers have developed a specialized Tajik text embedder by fine-tuning multilingual-e5-large-instruct on over 235,000 synthetic question-passage pairs. The resulting model significantly improves retrieval accuracy for Tajik laws and tax documents while remaining efficient enough to run on modest hardware.

What Problem Does This Paper Address?

Tajik, a language spoken by millions and written in a unique Cyrillic script, is often an "afterthought" in the world of Large Language Models (LLMs). The authors identify two major hurdles:

  1. Tokenization Gaps: Popular multilingual tokenizers (like those for E5 or BGE-M3) fail to recognize 4 out of the 12 unique Tajik letters (ҷ, ӣ, ӯ, ҷ), often collapsing them into a single "unknown" token.
  2. Lack of Benchmarks: There were no existing public benchmarks or models specifically designed for Tajik retrieval tasks, such as finding specific legal articles or tax regulations.

Methods

The authors started with the multilingual-e5-large-instruct (560M) model and applied a "recipe" for domain adaptation:

  • Synthetic Query Generation: Since human-labeled Tajik retrieval data is scarce, they used a Tajik-tuned LLM (Qwen3.6-27B) to generate questions for passages from Tajik laws, Wikipedia, and web text.
  • Contrastive Training: The model was trained using a multiple-negatives ranking loss. To make the task harder and the model sharper, they mined "hard negatives"—passages that look relevant but are actually incorrect—using an existing embedder.
  • Language Preservation (Replay): To ensure the model didn't "forget" how to handle English and Russian, they mixed in existing datasets from those languages (NQ and MIRACL) during training.
  • Precision Matters: Interestingly, the authors found that training in bfloat16 caused the model to collapse (loss dropped to zero), forcing them to use fp32 for stable training.
Figure 1: Top: training loss of Saidzoda-Tajik-E5-Large (logged every 10 steps). Bottom: two short test runs with the same settings; in bfloat16 the logged loss drops to exactly zero at step 8 and stays there, in fp32 it does not.

Datasets

The study utilized several Tajik-specific collections:

  • Tajik Law: 72,920 chunks of legal text.
  • Tajik Tax: 23,282 chunks of tax codes and guidance.
  • Tajik Wikipedia & Web Text: General knowledge and literature.
  • Belebele: A human-written reading comprehension dataset used for independent evaluation.
  • Replay Sets: English Natural Questions (NQ) and Russian MIRACL to maintain multilingual performance.

Key Insights and Ideas

  • The "Unknown" Token Tax: General multilingual models are effectively "blind" to certain Tajik nuances because their tokenizers weren't built with the language in mind.
  • Synthetic Data as a Bridge: A high-quality LLM in the target language can effectively "bootstrap" a retrieval model by creating massive amounts of training data where none existed.
  • Efficiency vs. Scale: You don't always need a multi-billion parameter model. A 560M model, when properly exposed to a low-resource language, can go toe-to-toe with 8B parameter giants.
  • Folding Fails: A common trick is "folding" unique Tajik letters into similar-looking Russian ones (e.g., mapping ҷ to ч). The authors tested this and found it actually decreased performance, suggesting the model benefits from learning the distinct Tajik characters.

Key Results

The fine-tuned model showed dramatic improvements over the base E5 model across several Tajik-specific test sets:

  • Law Retrieval: Recall@10 jumped from 80.2% to 93.6%.
  • Tax Retrieval: Recall@10 rose from 82.6% to 94.5%.
  • Human-Written Test (Belebele): Performance improved from 92.2% to 95.2%.
  • Efficiency: The model processes queries in ~12ms on an H100, roughly 3x faster than the 3B and 8B parameter competitors.

As shown in Table 3, the tuned 560M model (Ours) is competitive with much larger models like Giga-Embeddings-3B and Qwen3-Embedding-8B on Tajik tasks.

Table 3: Recall@10 (percent) of every embedder we scored on the same frozen test sets. Bold marks the highest value in each column, not a significant difference. Rows above the line are used as released; rows below were trained on our pairs (the LoRA runs used 40% of them). Belebele Russian and English check that the other languages are not lost.

Limitations and Future Work

  • Synthetic Bias: The law, tax, and Wikipedia test questions were generated by the same LLM used for training. This likely provides an "upper bound" on performance; the model might be learning the LLM's "style" rather than just the language.
  • Need for Human Data: The most critical next step is developing a retrieval benchmark where the questions are written by native Tajik speakers to validate these gains in real-world scenarios.
  • Scope: The study focused on retrieval only, not the subsequent "re-ranking" or "generation" steps of a full RAG (Retrieval-Augmented Generation) pipeline.

Summary updated

Abstract

Finding the passage that answers a question is the first step of most practical systems built on documents, and for Tajik we found no openly described model for it. We present Saidzoda-Tajik-E5-Large, a Tajik text embedder built from multilingual-e5-large-instruct (560M parameters). We asked our own Tajik language model to write questions for passages from Tajik laws, tax documents, Wikipedia and web text, and trained on 235,140 question and passage pairs for one epoch, with hard negatives chosen by another embedder and English and Russian pairs mixed in so the model would not forget those languages. On laws the model had never seen, the right passage is among the top ten results for 93.6% of questions, up from 80.2%; on tax documents, 94.5% against 82.6%. On Belebele, the one test we have that was written by people, the gain is smaller and not confirmed (92.2% to 95.2%, with touching intervals), and Russian and English do not change. On law and tax questions written in the same style as its training questions, it is tied with Giga-Embeddings-3B, a model five times its size; on Belebele the 3B and 8B models score slightly higher, and only after we fine-tuned them on our data are they clearly ahead. We also report what failed: training in bfloat16 broke the model, and replacing Tajik letters with similar Russian ones did not help. Our law, tax and Wikipedia test questions come from the same model that wrote the training questions, so we treat those scores as an upper limit until questions written by Tajik speakers are available.
BibTeX
@misc{saidzoda2026saidzoda,
  author = {Tohir Saidzoda and Nekruz Maxudov and Bedil Karimov and Abubakr Abdullozoda and Amin Jalilov and Jonibek Kishvarov and Rahmonali Sharipov and Mariyamkhon Shodyeva},
  title = {{Saidzoda-Tajik-E5-Large: Fine-Tuning a 560M Embedder for Tajik Retrieval}},
  year = {2026},
  eprint = {2611.00592},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00592},
}
Figure