Abstract
Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules.
Introduction
Large Language Models (LLMs) are incredibly capable, but they suffer from a massive "data desert" problem. While there are roughly 7,000 languages spoken globally, high-quality labeled datasets for complex tasks like summarization exist for only a handful (mostly English). Standard Supervised Fine-Tuning (SFT) is often prohibitively expensive, and even Parameter-Efficient Fine-Tuning (PEFT) typically requires labeled data in the target language to achieve high performance.
In this work, we present a method to "calculate" new model capabilities by performing simple addition and subtraction on the weights of parameter-efficient adapters, such as LoRA or Kronecker layers. By combining English task knowledge with target-language data through weight arithmetic, we achieve significant gains in zero-shot cross-lingual summarization without requiring any labeled data in the target language.
Methods: Language and Task Arithmetic
The core innovation is the use of expert modules that are trained separately and then combined arithmetically. We define two primary composition strategies:
Task-in-One-Language
When labeled data is available only in a source language $S$ (e.g., English) and unlabeled data is available for the target language $T$ (e.g., Portuguese), we create a new adapter $\theta_{task;T}$ using the formula: $$\theta_{task;T} = \lambda\theta_{task;S} + (1-\lambda)(\theta_{LM;T} - \theta_{LM;S})$$ where $\theta_{task;S}$ is an adapter trained on the source task, and $(\theta_{LM;T} - \theta_{LM;S})$ represents a "language shift" vector. Subtracting the source language adapter helps the model "unlearn" source-specific biases, facilitating better generation in the target language.
Task-in-Many-Languages
When labeled data is available in several languages, we can average the task adapters of the languages most linguistically related to the target (determined by the lang2vec toolkit) to create a more robust starting point.
Crucially, because these operations happen at the weight level before inference, the final model has the exact same inference cost as a standard model—there is no overhead for "composing" these experts at runtime.
Experimental Setup
We evaluated our approach on the XLSum benchmark, which includes news articles and summaries across 44 languages. We used mC4 for training language-only adapters and TyDi QA to test the generalizability of the method beyond summarization.
Results and Key Insights
Our evaluation on 11 "unseen" languages demonstrated that the "Add and Subtract" method consistently outperforms standard zero-shot baselines.
- Summarization Performance: In the Task-in-One-Language setup, average ROUGE-2 scores improved from 24.2 to 25.0. Leveraging multiple related languages (Task-in-Many-Languages) further increased performance to 26.0 ROUGE-2.
- Linguistic Proximity: Selective averaging based on linguistic distance is more effective than global averaging. Using the URIEL database to find syntactic and geographic neighbors prevents "negative transfer" from unrelated tongues.
- The Power of Subtraction: Subtracting the source language adapter is vital for generative tasks, as it helps steer the model away from English-specific biases.
Conclusion
Language and task arithmetic offers a scalable solution for zero-shot cross-lingual transfer. By treating PEFT modules as composable units, we can effectively extend LLMs to low-resource languages with minimal training and zero inference overhead. Future work will focus on automated hyperparameter tuning and dynamic expert selection.
Summary updated