Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization

Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, Priyanka Agrawal

Abstract

Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules.

Introduction

Large Language Models (LLMs) are incredibly capable, but they suffer from a massive "data desert" problem. While there are roughly 7,000 languages spoken globally, high-quality labeled datasets for complex tasks like summarization exist for only a handful (mostly English). Standard Supervised Fine-Tuning (SFT) is often prohibitively expensive, and even Parameter-Efficient Fine-Tuning (PEFT) typically requires labeled data in the target language to achieve high performance.

In this work, we present a method to "calculate" new model capabilities by performing simple addition and subtraction on the weights of parameter-efficient adapters, such as LoRA or Kronecker layers. By combining English task knowledge with target-language data through weight arithmetic, we achieve significant gains in zero-shot cross-lingual summarization without requiring any labeled data in the target language.

Methods: Language and Task Arithmetic

The core innovation is the use of expert modules that are trained separately and then combined arithmetically. We define two primary composition strategies:

Task-in-One-Language

When labeled data is available only in a source language $S$ (e.g., English) and unlabeled data is available for the target language $T$ (e.g., Portuguese), we create a new adapter $\theta_{task;T}$ using the formula: $$\theta_{task;T} = \lambda\theta_{task;S} + (1-\lambda)(\theta_{LM;T} - \theta_{LM;S})$$ where $\theta_{task;S}$ is an adapter trained on the source task, and $(\theta_{LM;T} - \theta_{LM;S})$ represents a "language shift" vector. Subtracting the source language adapter helps the model "unlearn" source-specific biases, facilitating better generation in the target language.

Task-in-Many-Languages

When labeled data is available in several languages, we can average the task adapters of the languages most linguistically related to the target (determined by the lang2vec toolkit) to create a more robust starting point.

Visual representation of Task-in-One-Language (a) and Task-in-Many-Languages (b) arithmetic composition.

Crucially, because these operations happen at the weight level before inference, the final model has the exact same inference cost as a standard model—there is no overhead for "composing" these experts at runtime.

Experimental Setup

We evaluated our approach on the XLSum benchmark, which includes news articles and summaries across 44 languages. We used mC4 for training language-only adapters and TyDi QA to test the generalizability of the method beyond summarization.

Results and Key Insights

Our evaluation on 11 "unseen" languages demonstrated that the "Add and Subtract" method consistently outperforms standard zero-shot baselines.

  • Summarization Performance: In the Task-in-One-Language setup, average ROUGE-2 scores improved from 24.2 to 25.0. Leveraging multiple related languages (Task-in-Many-Languages) further increased performance to 26.0 ROUGE-2.
  • Linguistic Proximity: Selective averaging based on linguistic distance is more effective than global averaging. Using the URIEL database to find syntactic and geographic neighbors prevents "negative transfer" from unrelated tongues.
  • The Power of Subtraction: Subtracting the source language adapter is vital for generative tasks, as it helps steer the model away from English-specific biases.
Heatmap illustrating zero-shot transfer performance (ROUGE-2 gains) across various training and evaluation language pairs.

Conclusion

Language and task arithmetic offers a scalable solution for zero-shot cross-lingual transfer. By treating PEFT modules as composable units, we can effectively extend LLMs to low-resource languages with minimal training and zero inference overhead. Future work will focus on automated hyperparameter tuning and dynamic expert selection.

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules.
BibTeX
@misc{chronopoulou2023language,
  author = {Alexandra Chronopoulou and Jonas Pfeiffer and Joshua Maynez and Xinyi Wang and Sebastian Ruder and Priyanka Agrawal},
  title = {{Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization}},
  year = {2023},
  eprint = {2311.09344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.09344},
}
Figure