Abstract
Instruction tuning has emerged as a powerful technique, significantly boosting zero-shot performance on unseen tasks. While recent work has explored cross-lingual generalization by applying instruction tuning to multilingual models, previous studies have primarily focused on English, with a limited exploration of non-English tasks. For an in-depth exploration of cross-lingual generalization in instruction tuning, we perform instruction tuning individually for two distinct language meta-datasets. Subsequently, we assess the performance on unseen tasks in a language different from the one used for training. To facilitate this investigation, we introduce a novel non-English meta-dataset named "KORANI" (Korean Natural Instruction), comprising 51 Korean benchmarks. Moreover, we design cross-lingual templates to mitigate discrepancies in language and instruction-format of the template between training and inference within the cross-lingual setting. Our experiments reveal consistent improvements through cross-lingual generalization in both English and Korean, outperforming baseline by average scores of 20.7% and 13.6%, respectively. Remarkably, these enhancements are comparable to those achieved by monolingual instruction tuning and even surpass them in some tasks. The result underscores the significance of relevant data acquisition across languages over linguistic congruence with unseen tasks during instruction tuning.
Introduction
Instruction tuning—the process of fine-tuning large language models (LLMs) on a diverse collection of tasks described via natural language prompts—has revolutionized zero-shot generalization. However, most research in this field has remained "English-centric." While multilingual models like mT5 exist, the community's understanding of how instruction-following skills transfer across linguistically diverse languages has been limited by a lack of high-quality, non-English datasets.
Previous cross-lingual studies often relied on machine-translated data or small, non-diverse datasets for non-English languages. This research addresses a critical question: Is it more effective to have a small amount of instruction data in the target language, or a massive amount of high-quality data in a different language? By comparing "monolingual" tuning (training and testing in the same language) against "cross-lingual" tuning (training in one, testing in another), this study reveals the underlying mechanics of cross-lingual task transfer.
The KORANI Dataset
A primary contribution of this work is the introduction of KORANI (Korean Natural Instruction). To move beyond the limitations of machine translation, KORANI was manually curated by experts to ensure high linguistic quality and diverse task coverage. It comprises 51 Korean benchmarks (34 NLU and 17 NLG tasks) across 17 task clusters. This dataset provides a rigorous testing ground for evaluating how well models can generalize instructions in a non-Western language.
Methodology
The researchers employed the mT5 (multilingual T5) model, with parameters ranging from 1.3B to 13B, as the backbone. The experimental framework focused on several key configurations:
- Cross-Lingual Instruction Tuning: Training on a meta-dataset in Language A (e.g., English P3) and evaluating on held-out tasks in Language B (e.g., Korean KORANI).
- Template Alignment: The authors identified "template misalignment" as a major barrier to performance. They argued that performance drops are often caused by differences in the style and format of instructions rather than the language itself. To address this, they introduced:
- CT (Cross-lingual Training): Using cross-lingual templates during the training phase.
- CI (Cross-lingual Inference): Using cross-lingual templates during evaluation to match the training style.
- Bilingual Training (mT-Bi): Training on both English and Korean datasets simultaneously to observe potential synergies.
Key Insights and Results
Task Logic Over Linguistic Congruence
The most striking finding is that learning how to perform a specific type of task (e.g., summarization or sentiment analysis) in English is highly transferable to Korean. The model appears to internalize the "logic" of the task, which transcends specific vocabulary. In several tasks, cross-lingual models (trained on English, tested on Korean) achieved performance nearly identical to—and sometimes better than—models trained directly on Korean.
The Importance of Formatting
Simply translating a prompt is insufficient for optimal transfer. Aligning the structural formatting (e.g., the placement of "Context" vs. "Question" labels) significantly boosts cross-lingual performance. Using Cross-lingual Inference (CI) templates—where the Korean test prompt was formatted to mirror the English training prompts—consistently yielded the highest scores.
Performance Gains and Scaling
Cross-lingual instruction tuning improved zero-shot performance by an average of 20.7% in English and 13.6% in Korean compared to the base mT5 model. Furthermore, as model size increased from 1.3B to 13B parameters, the gap between cross-lingual and monolingual performance narrowed. This suggests that larger models are more adept at internalizing cross-lingual task logic and overcoming linguistic barriers.
Bilingual Synergy
Training on both languages (mT-Bi) outperformed training on either language alone. This indicates that diverse data from multiple languages helps the model build a more robust, language-agnostic representation of instruction-following.
Conclusion
This study demonstrates that cross-lingual instruction tuning is a powerful and viable strategy, especially for languages where high-quality instruction data is scarce. By focusing on task relevance and template alignment, researchers can leverage high-resource languages like English to significantly boost the zero-shot capabilities of models in other languages. The introduction of KORANI provides a vital benchmark for future multilingual research, underscoring that the quality and diversity of task data are often more critical than linguistic congruence alone.
Summary updated