Adapting Qwen3 to Tajik from 0.6B to 8B: Lower Text Loss Did Not Mean Better Understanding

Tohir Saidzoda, Madina Abdusalomzoda, Nasimjon Rajabov, Akhrorkhon Ikromov, Komron Giyosiev, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon ShodyevaarXiv 2026 · arXiv

Adapting Large Language Models (LLMs) to low-resource languages is a common goal for regional AI labs. However, as a recent paper from the Saidzoda AI Lab demonstrates, "looking the part" isn't the same as "knowing the part." By adapting the Qwen3 family to Tajik, the researchers uncovered a stark disconnect between a model's fluency and its actual reading comprehension.

TL;DR

Lowering text loss (improving fluency) does not guarantee better reading comprehension in low-resource language adaptation. While lightweight LoRA adapters made models sound more Tajik, significant gains in understanding only appeared at the 8B parameter scale using a "heavy" recipe of continued pretraining and supervised fine-tuning.

What Problem Does This Paper Address?

Tajik, the state language of Tajikistan, uses a unique Cyrillic alphabet and is rarely supported by mainstream open-source models. For a small lab with limited compute (single GPUs), the challenge is twofold: how to effectively inject Tajik knowledge into existing models like Qwen3, and how to accurately measure if that adaptation actually worked.

The authors address a common pitfall in LLM development: relying on subjective "vibe checks" (reading model outputs) rather than objective benchmarks, which can lead researchers to believe a model has improved simply because it has become more fluent.

Methods

The researchers tested two distinct adaptation strategies across model sizes ranging from 0.6B to 8B parameters:

  1. The Light Recipe: Used small, attention-only LoRA adapters trained on roughly 21.4 million tokens of Tajik news and books. This was designed to be a low-compute "quick fix" for fluency.
  2. The Heavy Recipe: Involved Continued Pretraining (CPT) on a much larger corpus (~375 million tokens), followed by Supervised Fine-Tuning (SFT) on over a million Tajik chat examples.

Crucially, the authors performed a retrospective evaluation in 2026, scoring 15 trained models and 5 base models on a unified harness to see if their 2025 "progress" held up under scrutiny.

Datasets

  • Training: Tajik news articles, book chunks, and Tajik Wikipedia.
  • SFT: A "thinking-mode" dataset of ~1.2 million examples (though the authors later discovered only 0.5% actually contained reasoning).
  • Evaluation:
    • Belebele Tajik: 900 reading comprehension questions (the primary measure of understanding).
  • English MMLU: A subset of 1,003 questions used to measure "catastrophic forgetting" of English knowledge.
  • Tajik Text Loss: Measured over Belebele passages to quantify fluency.

Key Insights and Ideas

  • The Fluency Trap: Every model, regardless of size or recipe, showed a significant drop in Tajik text loss. They all became better at predicting the next Tajik token. However, for models at the 0.6B, 1.7B, and 4B scale, this did not translate to better scores on comprehension tests.
  • Scaling Thresholds: True understanding only emerged at the 8B scale. The 0.6B model, even with the "heavy" recipe, reached fluency levels comparable to the 8B model but remained at "chance" accuracy on multiple-choice questions.
  • Letter Preference Bias: The paper highlights a subtle evaluation error: small models often "improve" their scores not by understanding the text, but by shifting their bias toward a specific answer letter (e.g., always picking "A").
Figure 1: Tajik text loss (lower is better, axis reversed) against Belebele Tajik accuracy. The dotted line is chance. Every trained model has a lower text loss than its starting model; Belebele rose clearly only for the 8B models with continued pretraining. The 0.6B scores near chance mostly reflect a preference for one answer letter (Table 3).

Key Results

  • The 8B Success: The main model, e3, achieved a 11.4 point gain on Belebele Tajik over the stock Qwen3-8B, while maintaining stable English performance (only a 1.2 point drop on MMLU).
  • Small Model Failure: At the 0.6B scale, the heavy recipe lowered text loss by 1.88 nats but left comprehension scores near chance (~30%).
  • Catastrophic Forgetting: Smaller models paid a high price for their Tajik fluency; the 0.6B model's English MMLU score fell by up to 9.4 points during adaptation.

Limitations and Future Work

The authors are refreshingly honest about the "messy" reality of research. They acknowledge several limitations:

  • Lost Records: Some training logs and datasets were not properly documented or were deleted before they could be fully analyzed.
  • Lack of Control: The four groups of models differed in starting points and adapter sizes, making it hard to isolate a single variable for the 8B model's success.
  • Quantization Issues: A 4-bit release was found to be missing its transformer layers, rendering it unusable—a cautionary tale for automated release pipelines.

Future Directions: The authors recommend that future efforts should hold out test sets before training begins, measure comprehension and loss as separate metrics, and always check that quantized files actually load and score before public release.

Summary updated

Abstract

Small LoRA adapters trained on Tajik news and books lowered Qwen3's Tajik text loss at every size we tried, from 0.6B to 8B, by 0.60 to 1.45 nats per token, but none of them improved Tajik reading comprehension where it can be measured. On Belebele the 1.7B and 8B adapters left accuracy unchanged and the 4B adapter lowered it by 3.7 points; at 0.6B the score rose by 4.2 points but sits near chance and coincides with a change in which answer letter the model prefers, so we do not read it as understanding. At 8B, continued pretraining on about 375 million tokens of raw Tajik text was followed by a Belebele gain of 9.9 points. After a further supervised fine-tuning stage, our main model, e3, is 11.4 points above Qwen3-8B on Belebele; its English MMLU changed by -1.2 points, and a loss larger than 3.3 points is excluded. The same heavy recipe applied to Qwen3-0.6B-Base brought its text loss close to that of the 8B model after books-only pretraining, while its multiple-choice answers stayed near chance and its English MMLU fell by 7.6 points. We report 15 trained models and their 5 starting models from June to October 2025, scored in 2026 on one harness with paired intervals, and we report what went wrong: lost training records, a thinking-mode dataset with almost no reasoning in it, later branches whose training data was never documented, and a 4-bit release that contains no transformer layers.
BibTeX
@misc{saidzoda2026adapting,
  author = {Tohir Saidzoda and Madina Abdusalomzoda and Nasimjon Rajabov and Akhrorkhon Ikromov and Komron Giyosiev and Abubakr Abdullozoda and Amin Jalilov and Jonibek Kishvarov and Rahmonali Sharipov and Mariyamkhon Shodyeva},
  title = {{Adapting Qwen3 to Tajik from 0.6B to 8B: Lower Text Loss Did Not Mean Better Understanding}},
  year = {2026},
  eprint = {2611.06062},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06062},
}
Figure