TL;DR
Researchers from Saidzoda AI Lab demonstrate that while a model's tokenizer might "know" every letter of a language like Tajik, it doesn't mean it can "read" or retrieve information effectively without specific training. By fine-tuning the compact EmbeddingGemma-300M model, they achieved massive performance gains, making it competitive with models nearly twice its size while using 25% less storage space for vector embeddings.
What Problem Does This Paper Address?
In the world of Natural Language Processing, low-resource languages like Tajik often get left behind. Even when modern multilingual models include Tajik in their training data, their performance on specific tasks like information retrieval (finding the right document to answer a question) is often lackluster.
The authors highlight a curious case: EmbeddingGemma-300M. Unlike many other models whose tokenizers literally cannot "see" certain Tajik-specific Cyrillic letters (mapping them to an "unknown" token), EmbeddingGemma's tokenizer represents all 12 Tajik letter forms perfectly. Yet, out of the box, it performs poorly on Tajik retrieval tasks—significantly worse than models that are "blind" to some of the alphabet. This paper explores whether this small, efficient model can be "rescued" through targeted fine-tuning to match the performance of much larger counterparts.
Methods
The team applied a rigorous fine-tuning recipe to the 300M parameter model:
- Contrastive Learning: They used an "in-batch multiple-negatives loss" with gradient caching, a standard but effective technique for training dense retrievers.
- Hard Negative Mining: To make the model more robust, they didn't just give it correct pairs; they used a larger model (BGE-M3) to find "hard negatives"—passages that look relevant but are actually incorrect.
- Language Replay: To prevent the model from forgetting how to handle English and Russian while learning Tajik (a phenomenon called "catastrophic forgetting"), they included English and Russian "replay" data in the training mix.
- Precision Matters: Interestingly, the authors found that training in
bfloat16precision caused the loss to become "non-finite" (essentially breaking the math) very early on. They switched tofp32to ensure stable training.
Datasets
The authors utilized a diverse mix of Tajik-specific data for both training and evaluation:
- Training Data: A total of 235,140 pairs covering Tajik law, tax codes, Wikipedia, and general web/literature text.
- Synthetic Queries: Since human-written Tajik retrieval benchmarks are scarce, they used a large language model (Qwen3.6-27B) to generate questions based on passages.
- Belebele: A human-written, independent benchmark used to verify that the model's gains weren't just a result of learning the "style" of the synthetic question generator.
- Replay Sets: English Natural Questions (NQ) and Russian MIRACL were used to maintain multilingual capabilities.
Key Insights and Ideas
- Tokenizer Coverage $\neq$ Performance: The most striking "aha moment" is that having a complete vocabulary for a language's alphabet is a precondition for learning, not a guarantee of performance. A model can know the letters but still fail to understand the relationships between questions and answers in that language.
- Efficiency Gains: The 300M model produces 768-dimensional vectors, whereas the larger 560M models produce 1024-dimensional vectors. This means the 300M model's search index takes up 25% less space, making it a prime candidate for deployment on modest hardware or mobile devices.
- Rapid Convergence: The model learned incredibly fast. A checkpoint taken at just step 100 (out of 466) had already captured 88% of the total gain on the law retrieval task.
Key Results
The fine-tuning was remarkably successful, transforming the model from "weak" to "competitive":
- Massive Tajik Gains: On the Tajik Law test, Recall@10 jumped from 49.3% to 92.4% (+43.1 points). On the human-written Belebele Tajik set, it rose from 69.8% to 92.6%.
- Closing the Gap: After training, the 300M model performed within roughly 1-3 points of models twice its size (like E5-Large and BGE-M3) on most Tajik tasks.
- The Cost of Learning: There was a small "tax" for this Tajik expertise; the model lost about 3 points of Recall@1 in Russian and English, suggesting that even with replay data, balancing multiple languages in a small parameter space is a challenge.
Limitations and Future Work
While the results are promising, the authors note several caveats:
- Upper Bounds: Because many test questions were generated by the same type of model used for training, these scores should be viewed as an "upper limit" of performance.
- Small Samples: Some test sets, like the Tax set (109 questions), are quite small, leading to wider statistical intervals.
- Real-world Testing: The authors did not test the model's latency on actual edge devices, which is the primary use case for a model of this size.
- Informal Language: The study focused on formal documents (laws, Wikipedia); performance on spoken or informal Tajik remains untested.
The researchers have released the model weights and training logs on Hugging Face, providing a valuable resource for the Tajik NLP community.
Summary updated