Saidzoda-Kazakh-EmbeddingGemma-300M: How Far One Epoch of Generated Questions Takes a Small Kazakh Embedder

Tohir Saidzoda, Madina Abdusalomzoda, Abubakr Abdullozoda, Amin Jalilov, Faroiz Bobokhon, Aibek ShokanulyarXiv 2026 · arXiv

For many low-resource languages, the gap between "state-of-the-art" and "actually useful" is often a matter of data. In their recent paper, researchers from Saidzoda AI Lab demonstrate that you don't necessarily need massive human-labeled datasets to bridge this gap. By fine-tuning a small, 300M-parameter model on synthetic data for just one epoch, they significantly boosted Kazakh retrieval performance, outperforming models many times its size.

TL;DR

The authors present Saidzoda-Kazakh-EmbeddingGemma-300M, a small open-source embedder fine-tuned for one epoch on ~208k pairs of synthetic Kazakh questions and source passages. This targeted training boosted Recall@1 on the Belebele benchmark from 60.8% to 78.0%, outperforming much larger models and proprietary APIs like OpenAI’s text-embedding-3-large.

What Problem Does This Paper Address?

While massive multilingual models exist, their performance on specific languages like Kazakh—a Turkic language written in Cyrillic—often lags behind. High-quality, human-annotated retrieval datasets for Kazakh are scarce. The authors wanted to see if a small, efficient model (EmbeddingGemma-300M) could be transformed into a high-performing Kazakh specialist using a pipeline of generated questions, and how such a model would stack up against the giants on the Massive Text Embedding Benchmark (MTEB).

Methods

The team employed a "teacher-student" synthetic data pipeline:

  1. Generation: An in-house 27B Qwen model acted as the "teacher," writing questions based on Kazakh source passages from Wikipedia, news, and books.
  2. Verification: A second model (Qwen3.6-27B) checked these questions to ensure they were actually answerable by the passage.
  3. Fine-tuning: They performed full fine-tuning on the 300M-parameter EmbeddingGemma model for a single epoch.
  4. Loss & Negatives: They used cached multiple-negatives ranking loss and mined "hard negatives" (plausible but incorrect passages) using the BGE-M3 model to sharpen the model's ability to distinguish between similar texts.
  5. Replay: To prevent the model from forgetting other languages, they included "replay" data in English and Russian.
Training loss over 400 steps shows consistent convergence.

Datasets

The training set consisted of 208,511 pairs:

  • 166,263 Kazakh pairs: Synthetic questions grounded in real source passages.
  • English & Russian Replay: ~42k pairs from Natural Questions and MIRACL to maintain multilingual capabilities.

Evaluation was conducted on:

  • Belebele (Kazakh): A human-written reading comprehension dataset (900 questions). This served as the primary "honest" test since it wasn't generated by their pipeline.
  • Kazakh Wikipedia: 500 synthetic questions on held-out articles.
  • Cross-lingual sets: Belebele versions in Tajik, Uzbek, and Kyrgyz.

Key Insights and Ideas

  • Synthetic Data as a Force Multiplier: High-quality synthetic data, when verified by a second model, can substitute for expensive human labeling. One epoch was enough to move the needle significantly.
  • Small Models Punch Up: Despite having only 308M parameters, the model outperformed 161 out of 194 models on the MTEB leaderboard for this task, including the API-based text-embedding-3-large (which scored only 47.1% Recall@1).
  • The Transfer Paradox: Training on Kazakh improved performance on related Turkic languages (Uzbek and Kyrgyz). However, the authors found that their Tajik-only model (a non-Turkic language) provided similar gains. This suggests that the improvement might stem from general "retrieval task" adaptation or shared regional vocabulary rather than specific linguistic transfer from Kazakh.
Comparison of Recall@1 across different languages and models.

Key Results

  • Belebele Recall@1: Jumped from 60.8% to 78.0%.
  • Belebele Recall@10: Reached 91.4%.
  • MTEB Standing: The model is now among the top performers for Kazakh. While it trails a few massive multi-billion parameter models (like BGE-M3), it is the leader among models in its size class.
Table 2: Kazakh retrieval in percent. Belebele: each of the 900 questions must find its own passage among 488 Kazakh passages; the dataset is fully parallel across languages. Wikipedia: generated questions on held-out articles over 30,000 passages, written by the same generator family as our training questions, so the values are optimistic. Intervals are 95% bootstrap intervals for recall@10. Our model is the row in bold. The Tajik model and the four-language model are our earlier EmbeddingGemma fine-tunes; Saidzoda-Tajik-E5-Large is our Tajik E5 model. Published scores of further models are in Table 4.

Limitations and Future Work

  • Generator Bias: The Wikipedia test results are likely optimistic because the test questions were generated by the same family of models used for training.
  • Contamination: It is difficult to prove that the "teacher" models hadn't seen parts of the Belebele passages during their own pre-training.
  • Single Epoch: The study only explored one epoch of training; the effects of longer training or different "replay" ratios remain unknown.
  • Benchmark Scope: The authors did not evaluate on other Kazakh-specific benchmarks like KazQAD, leaving room for broader validation.

The model and its training logs have been released on the Hugging Face Hub (Tohirju/Saidzoda-Kazakh-EmbeddingGemma-300M), providing a new resource for Central Asian NLP.

Summary updated

Abstract

EmbeddingGemma-300M is a small open embedder, and used as released it finds the right passage first for 60.8% of the Kazakh Belebele retrieval questions. We present Saidzoda-Kazakh-EmbeddingGemma-300M, the same model fully fine-tuned for one epoch on 208,511 pairs: 166,263 Kazakh questions that our own language model wrote about source passages from encyclopedias, Wikipedia, news and books, each checked by a second model, plus English and Russian replay. On Kazakh Belebele, recall@1 rises from 60.8% to 78.0% and recall@10 from 76.3% to 91.4%. Our model scores above 161 of the 194 models with a published MTEB score on the same test, among them OpenAI text-embedding-3-large (47.1%). The other 33 score higher: the ten best are large multi-billion-parameter or closed models, reaching up to 92.2%, and at least 9 are open models of up to 0.6B parameters, the smallest with 339M. We also scored 6 Kazakh-specific embedders shared on Hugging Face with the same code in one run: our model (77.3% there) is ahead of 4 of them, including the two E5-base models of similar size (72.6% and 69.0%), and behind kaz-embedder (86.0%, three towers with 1,695M parameters in total) and a BGE-M3 fine-tune (81.0%); at recall@10 its interval overlaps those of its nearest neighbours on both sides. Our four-language model, trained with a capped share of the same Kazakh pairs, is 2.8 points lower at recall@1, with overlapping recall@10 intervals. The Kazakh model also scores higher than the released model on the Uzbek and Kyrgyz versions of Belebele, but our Tajik model, trained without any Turkic pairs, gains as much there, so these gains are not specific to Kazakh training. Our Kazakh Wikipedia test questions come from the same generator family as the training questions, so we rest the main claims on Belebele.
BibTeX
@misc{saidzoda2026saidzoda,
  author = {Tohir Saidzoda and Madina Abdusalomzoda and Abubakr Abdullozoda and Amin Jalilov and Faroiz Bobokhon and Aibek Shokanuly},
  title = {{Saidzoda-Kazakh-EmbeddingGemma-300M: How Far One Epoch of Generated Questions Takes a Small Kazakh Embedder}},
  year = {2026},
  eprint = {2611.16399},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16399},
}
Figure