This report details the development of Saidzoda-Tajik-LFM-2.6B, a specialized adaptation of Liquid AI’s hybrid model designed to bring tool-calling capabilities to the Tajik language. The authors demonstrate that while continued pretraining on raw text is necessary for low-resource language adaptation, it comes with a significant "tax" on specialized skills that must be carefully recovered through fine-tuning.
TL;DR
The authors adapted a 2.6B parameter hybrid model to Tajik tool calling, raising accuracy from 20.69% to 64.14% while maintaining English performance. A key finding is that continued pretraining (CPT) on raw text temporarily "breaks" specialized skills like tool calling, which then require targeted supervised fine-tuning (SFT) with multi-lingual replay to restore.
What Problem Does This Paper Address?
Tajik, a Cyrillic-script language with unique characters (ғ, ӣ, қ, ӯ, ҳ, ҷ), is rarely supported by open-source LLMs. While large models exist for general Tajik chat, there is a gap for small, efficient models that can run on modest hardware to perform tool calling—the ability to turn a natural language request into a structured API call.
Existing models like LFM2.5-2.6B perform well in English tool calling (~76%) but fail significantly when prompted in Tajik (~21%). The challenge lies in teaching the model a new language without losing the complex reasoning required for function calling.
Methods
The researchers employed a two-stage adaptation recipe using LoRA (Low-Rank Adaptation) with rank-stabilized scaling:
- Continued Pretraining (CPT): The model was trained for 3 epochs on ~2.43 billion tokens of Tajik web text. This stage was intended to teach the model the Tajik language.
- Supervised Fine-Tuning (SFT): The model was then trained on a specific "mix" designed to balance new language acquisition with skill retention:
- 40% Tajik Tool Calls: English datasets (Glaive, Hermes, ToolACE) machine-translated into Tajik.
- 35% English Replay: The original English tool-calling data to prevent catastrophic forgetting.
- 25% Tajik Instructions: General instruction data to maintain conversational ability in Tajik.
To evaluate the model, they used both strict parsing (matching the exact native format) and lenient parsing (accepting JSON or XML formats) to distinguish between the model forgetting a format versus forgetting the skill itself.
Datasets
- Pretraining: A massive crawl of Tajik web documents (1.48M documents) from sources like HPLT 2.0, FineWeb-2, and Tajik Wikipedia.
- Fine-tuning: 176,532 rows of tool-calling and instruction data.
- Evaluation: The authors introduced a new Tajik tool-calling test set consisting of 174 items adapted from the Berkeley Function Calling Leaderboard (BFCL) v3, with user turns machine-translated into Tajik while keeping API schemas in English.
Key Insights and Ideas
- The CPT "Skill Tax": Raw-text CPT is a double-edged sword. While it improved Tajik knowledge, it initially drove tool-calling accuracy to zero. Lenient parsing revealed that CPT didn't just break the output format; it actively degraded the underlying logic of selecting the right tool.
- Recovery via Replay: The SFT stage successfully restored the tool-calling skill. Crucially, including English "replay" data ensured that the model's original English capabilities remained intact (76.55% base vs 77.24% final).
- Tokenizer Readiness: A vital first step was confirming the tokenizer could represent Tajik's unique letters. Fortunately, the LFM2.5 tokenizer already had single-token representations for these characters, though Tajik text remains roughly twice as "expensive" in tokens per word compared to English.
- Format vs. Skill: By using a lenient parser, the authors could see that the base model actually "knew" more Tajik than the strict scores suggested—it just often answered in non-standard formats (like XML or JSON) that a strict production parser would reject.
Key Results
The primary success was the massive jump in Tajik tool-calling performance.
- Tajik Tool Calling: Strict accuracy rose from 20.69% to 64.14%.
- English Tool Calling: Remained stable, moving from 76.55% to 77.24%.
- General Knowledge: Tajik knowledge benchmark scores improved from 28.0 to 35.0. However, this came at the cost of English general knowledge, with English MMLU dropping from 60.8 to 52.5.
The progression of accuracy across training stages is clearly visible in the authors' evaluation:
The breakdown by category shows that while single and parallel calls improved significantly, multi-turn conversations remain a challenge for a model of this size:
Limitations and Future Work
- Abstention Weakness: The model struggles to say "no." If no tool fits the request, the model often tries to force a call anyway (irrelevance accuracy dropped to 6.9% in Tajik). The authors attribute this to a lack of "no-call" examples in the SFT data.
- Multi-turn Logic: Performance on multi-turn tool calling remains low (~17% in Tajik), suggesting that 2.6B parameters may be tight for complex stateful reasoning.
- Machine Translation Bias: The test set relies on machine translation. Real-world Tajik, with its various dialects and informal spellings, may yield different results.
- Future Directions: The team plans to explore preference or reinforcement training (RLHF/DPO) to address the abstention and multi-turn weaknesses.
Summary updated