Abstract
The introduction of pretrained cross-lingual language models brought decisive improvements to multilingual NLP tasks. However, the lack of labelled task data necessitates a variety of methods aiming to close the gap to high-resource languages. Zero-shot methods in particular, often use translated task data as a training signal to bridge the performance gap between the source and target language(s). We introduce XeroAlign, a simple method for task-specific alignment of cross-lingual pretrained transformers such as XLM-R. XeroAlign uses translated task data to encourage the model to generate similar sentence embeddings for different languages. The XeroAligned XLM-R, called XLM-RA, shows strong improvements over the baseline models to achieve state-of-the-art zero-shot results on three multilingual natural language understanding tasks. XLM-RA's text classification accuracy exceeds that of XLM-R trained with labelled data and performs on par with state-of-the-art models on a cross-lingual adversarial paraphrasing task.
Introduction
Pretrained cross-lingual language models (PXLMs) like XLM-R are the backbone of modern multilingual NLP. While they possess knowledge of dozens of languages, a significant barrier remains: high-quality labeled data for specific tasks (like intent classification or slot filling) is typically only available in English. When researchers attempt to apply an English-trained model to a target language like Thai or Hindi—a process known as zero-shot transfer—a massive performance gap often emerges.
Previous attempts to bridge this gap have relied on massive parallel corpora or complex contrastive learning setups requiring large batches of negative samples. XeroAlign proposes a simpler, more efficient alternative: task-specific alignment using only translated data.
The XeroAlign Method
The core of XeroAlign is an auxiliary alignment loss function integrated into the training loop. Instead of just training the model to predict a label for the primary task, the authors employ joint optimization (multi-task learning) to force the model to achieve two goals simultaneously:
- Task Loss: Predict the correct label for an English sentence using standard Cross-Entropy.
- Alignment Loss: Ensure the sentence embedding (the
[CLS]token) of the English sentence is as close as possible to the embedding of its machine-translated counterpart in a target language.
XeroAlign uses Mean Squared Error (MSE) to minimize the distance between these two embeddings. This joint optimization is critical; if alignment and task training are done sequentially, the model suffers from catastrophic forgetting, losing either its task-specific logic or its cross-lingual alignment. Unlike contrastive learning, XeroAlign only requires "positive" translation pairs, making it computationally efficient and suitable for smaller, task-specific datasets.
Experimental Setup
The authors evaluated XeroAlign on four major multilingual benchmarks covering 11 unique languages, including German, Spanish, French, Thai, Hindi, Japanese, Korean, Chinese, Turkish, and Portuguese:
- PAWS-X: Paraphrase identification.
- MTOD & MTOP: Multilingual task-oriented datasets for intent classification and slot filling.
- MultiATIS++: An extension of the classic airline travel information dataset.
Key Results and Insights
XLM-RA (the XeroAligned version of XLM-R) achieved state-of-the-art (SOTA) or near-SOTA results on three out of four datasets. A particularly striking finding is the "Transfer Advantage": XeroAligned models often outperform "Target" models that were trained on actual gold-standard labels in the target language. This suggests that the high-resource signal from English provides a cleaner learning path, which XeroAlign then successfully ports to other languages.
As shown in the results, the gains are most dramatic for low-resource languages like Thai (th) and Hindi (hi). Furthermore, the model demonstrates impressive cross-lingual generalization; aligning the model on one language pair (e.g., English to German) often improves performance on unrelated languages (e.g., Japanese) that were not part of the alignment training. This indicates that the model is learning a more universal, language-agnostic representation space.
Limitations and Future Work
While XeroAlign dominates in classification tasks, it still faces challenges with Structured Prediction (such as slot filling). There remains an 8-19 point gap between zero-shot performance and target-language training for these tasks. This is likely because slot filling relies heavily on syntax and word order, which vary significantly between languages, making simple sentence-level [CLS] alignment less effective.
Future work aims to explore whether aligning individual token representations, rather than just the sentence-level embedding, can close the gap in structured prediction. Additionally, the authors plan to apply XeroAlign to more complex tasks like Question Answering (Q&A) and Named Entity Recognition (NER).
Summary updated