Adapting Large Language Models (LLMs) to low-resource languages is a common goal for regional AI labs. However, as a recent paper from the Saidzoda AI Lab demonstrates, "looking the part" isn't the same as "knowing the part." By adapting the Qwen3 family to Tajik, the researchers uncovered a stark disconnect between a model's fluency and its actual reading comprehension.
TL;DR
Lowering text loss (improving fluency) does not guarantee better reading comprehension in low-resource language adaptation. While lightweight LoRA adapters made models sound more Tajik, significant gains in understanding only appeared at the 8B parameter scale using a "heavy" recipe of continued pretraining and supervised fine-tuning.
What Problem Does This Paper Address?
Tajik, the state language of Tajikistan, uses a unique Cyrillic alphabet and is rarely supported by mainstream open-source models. For a small lab with limited compute (single GPUs), the challenge is twofold: how to effectively inject Tajik knowledge into existing models like Qwen3, and how to accurately measure if that adaptation actually worked.
The authors address a common pitfall in LLM development: relying on subjective "vibe checks" (reading model outputs) rather than objective benchmarks, which can lead researchers to believe a model has improved simply because it has become more fluent.
Methods
The researchers tested two distinct adaptation strategies across model sizes ranging from 0.6B to 8B parameters:
- The Light Recipe: Used small, attention-only LoRA adapters trained on roughly 21.4 million tokens of Tajik news and books. This was designed to be a low-compute "quick fix" for fluency.
- The Heavy Recipe: Involved Continued Pretraining (CPT) on a much larger corpus (~375 million tokens), followed by Supervised Fine-Tuning (SFT) on over a million Tajik chat examples.
Crucially, the authors performed a retrospective evaluation in 2026, scoring 15 trained models and 5 base models on a unified harness to see if their 2025 "progress" held up under scrutiny.
Datasets
- Training: Tajik news articles, book chunks, and Tajik Wikipedia.
- SFT: A "thinking-mode" dataset of ~1.2 million examples (though the authors later discovered only 0.5% actually contained reasoning).
- Evaluation:
- Belebele Tajik: 900 reading comprehension questions (the primary measure of understanding).
- English MMLU: A subset of 1,003 questions used to measure "catastrophic forgetting" of English knowledge.
- Tajik Text Loss: Measured over Belebele passages to quantify fluency.
Key Insights and Ideas
- The Fluency Trap: Every model, regardless of size or recipe, showed a significant drop in Tajik text loss. They all became better at predicting the next Tajik token. However, for models at the 0.6B, 1.7B, and 4B scale, this did not translate to better scores on comprehension tests.
- Scaling Thresholds: True understanding only emerged at the 8B scale. The 0.6B model, even with the "heavy" recipe, reached fluency levels comparable to the 8B model but remained at "chance" accuracy on multiple-choice questions.
- Letter Preference Bias: The paper highlights a subtle evaluation error: small models often "improve" their scores not by understanding the text, but by shifting their bias toward a specific answer letter (e.g., always picking "A").
Key Results
- The 8B Success: The main model, e3, achieved a 11.4 point gain on Belebele Tajik over the stock Qwen3-8B, while maintaining stable English performance (only a 1.2 point drop on MMLU).
- Small Model Failure: At the 0.6B scale, the heavy recipe lowered text loss by 1.88 nats but left comprehension scores near chance (~30%).
- Catastrophic Forgetting: Smaller models paid a high price for their Tajik fluency; the 0.6B model's English MMLU score fell by up to 9.4 points during adaptation.
Limitations and Future Work
The authors are refreshingly honest about the "messy" reality of research. They acknowledge several limitations:
- Lost Records: Some training logs and datasets were not properly documented or were deleted before they could be fully analyzed.
- Lack of Control: The four groups of models differed in starting points and adapter sizes, making it hard to isolate a single variable for the 8B model's success.
- Quantization Issues: A 4-bit release was found to be missing its transformer layers, rendering it unusable—a cautionary tale for automated release pipelines.
Future Directions: The authors recommend that future efforts should hold out test sets before training begins, measure comprehension and loss as separate metrics, and always check that quantized files actually load and score before public release.
Summary updated