TL;DR
Researchers from Saidzoda AI Lab have developed a single, compact 300M-parameter text embedding model that effectively serves four Central Asian languages: Tajik, Kazakh, Uzbek, and Kyrgyz. By fine-tuning with a mix of synthetic "grounded" questions and "replay" data, they achieved massive performance gains (up to +26 points) over base models, proving that one small model can efficiently handle a linguistically diverse region.
What Problem Does This Paper Address?
Building retrieval systems for Central Asia is a logistical and computational headache. The region uses a mix of language families—Tajik (Persian/Indo-European) and Kazakh, Uzbek, and Kyrgyz (Turkic)—written in different scripts (Cyrillic and Latin).
Standard practice often involves serving one specialized model per language, which quadruples the infrastructure costs. Meanwhile, small, open-source "off-the-shelf" embedders like Google's EmbeddingGemma-300M perform poorly on these languages, often failing to find the correct passage more than half the time. The authors set out to see if a single, tiny model could serve the entire region without sacrificing the performance of language-specific models.
Methods
The team started with a version of EmbeddingGemma-300M already fine-tuned for Tajik. They then performed one additional epoch of training using a carefully curated mix of data:
- Synthetic Data Generation: They used a 27B-parameter Qwen model to write "grounded" questions based on source passages (Wikipedia, news, etc.) in Kazakh, Uzbek, and Kyrgyz. A second model then verified these pairs to ensure accuracy.
- Training Recipe: They employed in-batch multiple-negatives loss with gradient caching, allowing for larger effective batch sizes on a single GPU. They also mined "hard negatives"—passages that are similar to the correct one but don't actually answer the question—to sharpen the model's retrieval skills.
- Catastrophic Forgetting Prevention: To ensure the model didn't "forget" Tajik or its original English/Russian capabilities, they included "refresher" Tajik data and "replay" sets from English and Russian benchmarks.
Datasets
The training set consisted of 279,904 pairs, broken down as follows:
For evaluation, the authors used:
- Belebele: A parallel reading comprehension dataset used as a retrieval benchmark.
- Wikipedia Test Sets: 500 generated questions per Turkic language on held-out articles.
- Domain-Specific Tajik Sets: Specialized sets for Tajik law and tax documents.
Key Insights and Ideas
- The "One Model" Success: The unified model performed nearly as well as specialized single-language models (within 2.8 points). This suggests that for many practical applications, the efficiency of serving one model outweighs the marginal gains of language-specific ones.
- Cross-Lingual Transfer: Interestingly, training on Tajik alone improved performance on the Turkic languages. This is likely due to shared cultural context, loanwords, and the model learning the general "style" of retrieval tasks.
- Small Models vs. Giants: While the 300M model still trails larger 560M models (like XLM-RoBERTa based E5) on general benchmarks, it actually outperforms them on specific local domains like Tajik law. This highlights the power of targeted fine-tuning for low-resource languages.
Key Results
The model, named Saidzoda-CA-EmbeddingGemma-300M, showed dramatic improvements over the base EmbeddingGemma-300M model.
- Belebele Recall@1: Gains ranged from +14.4 to +26.0 points per language, reaching a final accuracy of ~75-78%.
- Mean Performance: The average recall across all four languages jumped from 58.0% to 76.0%.
- Tajik Domain Retention: The model successfully maintained its high performance on Tajik legal documents (92.2% recall@10), proving the "refresher" strategy worked.
Limitations and Future Work
- Synthetic Bias: Much of the evaluation data was generated by the same family of models used for training. The authors acknowledge these scores should be viewed as "upper bounds."
- The Turkic Gap: The model still trails larger 560M-parameter models on Turkic language benchmarks by about 10 points. Closing this gap while staying at the 300M scale remains a challenge.
- Script Coverage: The Uzbek evaluation was almost entirely in Latin script; the model's performance on Cyrillic Uzbek remains largely untested.
- Future Directions: The authors suggest testing with real-world user queries and exploring why the performance gap between small and large models persists more strongly in Turkic languages than in Tajik.
Summary updated