Saidzoda-Tajik-EmbeddingGemma-300M: Knowing Every Letter Is Not Reading Tajik

Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon ShodyevaarXiv 2026 · arXiv

TL;DR

Researchers from Saidzoda AI Lab demonstrate that while a model's tokenizer might "know" every letter of a language like Tajik, it doesn't mean it can "read" or retrieve information effectively without specific training. By fine-tuning the compact EmbeddingGemma-300M model, they achieved massive performance gains, making it competitive with models nearly twice its size while using 25% less storage space for vector embeddings.

What Problem Does This Paper Address?

In the world of Natural Language Processing, low-resource languages like Tajik often get left behind. Even when modern multilingual models include Tajik in their training data, their performance on specific tasks like information retrieval (finding the right document to answer a question) is often lackluster.

The authors highlight a curious case: EmbeddingGemma-300M. Unlike many other models whose tokenizers literally cannot "see" certain Tajik-specific Cyrillic letters (mapping them to an "unknown" token), EmbeddingGemma's tokenizer represents all 12 Tajik letter forms perfectly. Yet, out of the box, it performs poorly on Tajik retrieval tasks—significantly worse than models that are "blind" to some of the alphabet. This paper explores whether this small, efficient model can be "rescued" through targeted fine-tuning to match the performance of much larger counterparts.

Methods

The team applied a rigorous fine-tuning recipe to the 300M parameter model:

  • Contrastive Learning: They used an "in-batch multiple-negatives loss" with gradient caching, a standard but effective technique for training dense retrievers.
  • Hard Negative Mining: To make the model more robust, they didn't just give it correct pairs; they used a larger model (BGE-M3) to find "hard negatives"—passages that look relevant but are actually incorrect.
  • Language Replay: To prevent the model from forgetting how to handle English and Russian while learning Tajik (a phenomenon called "catastrophic forgetting"), they included English and Russian "replay" data in the training mix.
  • Precision Matters: Interestingly, the authors found that training in bfloat16 precision caused the loss to become "non-finite" (essentially breaking the math) very early on. They switched to fp32 to ensure stable training.
The training loss curves show all three models flattening out within a single epoch, though they start from different baselines.

Datasets

The authors utilized a diverse mix of Tajik-specific data for both training and evaluation:

  • Training Data: A total of 235,140 pairs covering Tajik law, tax codes, Wikipedia, and general web/literature text.
  • Synthetic Queries: Since human-written Tajik retrieval benchmarks are scarce, they used a large language model (Qwen3.6-27B) to generate questions based on passages.
  • Belebele: A human-written, independent benchmark used to verify that the model's gains weren't just a result of learning the "style" of the synthetic question generator.
  • Replay Sets: English Natural Questions (NQ) and Russian MIRACL were used to maintain multilingual capabilities.
Breakdown of training pairs by domain.

Key Insights and Ideas

  • Tokenizer Coverage $\neq$ Performance: The most striking "aha moment" is that having a complete vocabulary for a language's alphabet is a precondition for learning, not a guarantee of performance. A model can know the letters but still fail to understand the relationships between questions and answers in that language.
  • Efficiency Gains: The 300M model produces 768-dimensional vectors, whereas the larger 560M models produce 1024-dimensional vectors. This means the 300M model's search index takes up 25% less space, making it a prime candidate for deployment on modest hardware or mobile devices.
  • Rapid Convergence: The model learned incredibly fast. A checkpoint taken at just step 100 (out of 466) had already captured 88% of the total gain on the law retrieval task.

Key Results

The fine-tuning was remarkably successful, transforming the model from "weak" to "competitive":

  • Massive Tajik Gains: On the Tajik Law test, Recall@10 jumped from 49.3% to 92.4% (+43.1 points). On the human-written Belebele Tajik set, it rose from 69.8% to 92.6%.
  • Closing the Gap: After training, the 300M model performed within roughly 1-3 points of models twice its size (like E5-Large and BGE-M3) on most Tajik tasks.
  • The Cost of Learning: There was a small "tax" for this Tajik expertise; the model lost about 3 points of Recall@1 in Russian and English, suggesting that even with replay data, balancing multiple languages in a small parameter space is a challenge.
Table 2: Recall@10 (top block, with 95% bootstrap intervals) and recall@1 (bottom block) in percent, for EmbeddingGemma-300M as released and after our fine-tuning. Change is in percentage points. Law, tax and Wikipedia queries were written by an LLM on held-out documents; only the Belebele sets are human-written.

Limitations and Future Work

While the results are promising, the authors note several caveats:

  • Upper Bounds: Because many test questions were generated by the same type of model used for training, these scores should be viewed as an "upper limit" of performance.
  • Small Samples: Some test sets, like the Tax set (109 questions), are quite small, leading to wider statistical intervals.
  • Real-world Testing: The authors did not test the model's latency on actual edge devices, which is the primary use case for a model of this size.
  • Informal Language: The study focused on formal documents (laws, Wikipedia); performance on spoken or informal Tajik remains untested.

The researchers have released the model weights and training logs on Hugging Face, providing a valuable resource for the Tajik NLP community.

Summary updated

Abstract

EmbeddingGemma-300M is a small open embedder whose tokenizer represents every Tajik letter, and as released it is weak at Tajik retrieval: on our Tajik law test the right passage is in its top ten results for 49.3% of questions, on tax documents for 45.9%, and on the human-written Belebele Tajik set for 69.8%, against 98.6% on the Russian version of the same set. We present Saidzoda-Tajik-EmbeddingGemma-300M, the same model fully fine-tuned for one epoch on 235,140 question and passage pairs written by our own Tajik language model for laws, tax documents, Wikipedia and web text, with the same mined hard negatives, English and Russian replay, loss, global batch, learning rate and epoch as our earlier Saidzoda-Tajik-E5-Large (arXiv:2611.00592) and BGE-M3 models. After training, law reaches 92.4% and tax 91.7%; on Belebele, the one test written by people, recall@10 rises from 69.8% to 92.6% and the first result is right for 76.6% of questions instead of 51.7%. A checkpoint at step 100 of 466 already holds 88% of the law gain at recall@10. At recall@10 the result is no more than 2.8 points below the two 560M models trained on the same pairs on law, tax and Wikipedia, within the intervals, with 768-dimensional vectors and about half their parameters; at recall@1, for which we have no intervals, it stays 7.0 points behind the E5 model on Belebele. The cost is a drop of 3.0 and 2.8 points in Russian and English recall@1, larger than for the two earlier models. We also report that, according to our run notes, a bfloat16 test run produced a non-finite loss at step 3 which the training framework logged as zero, so training was done in fp32. Our generated test questions come from the same model as the training questions, so we treat those scores as an upper limit until questions written by Tajik speakers exist.
BibTeX
@misc{saidzoda2026saidzoda,
  author = {Tohir Saidzoda and Nekruz Maxudov and Bedil Karimov and Abubakr Abdullozoda and Amin Jalilov and Jonibek Kishvarov and Rahmonali Sharipov and Mariyamkhon Shodyeva},
  title = {{Saidzoda-Tajik-EmbeddingGemma-300M: Knowing Every Letter Is Not Reading Tajik}},
  year = {2026},
  eprint = {2611.11243},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11243},
}
Figure