Zero-shot Cross-lingual Transfer without Parallel Corpus

Yuyang Zhang, Xiaofeng Han, Baojun WangarXiv 2023 · OpenReview

Abstract

Recently, although pre-trained language models have achieved great success on multilingual NLP (Natural Language Processing) tasks, the lack of training data on many tasks in low-resource languages still limits their performance. One effective way of solving that problem is to transfer knowledge from rich-resource languages to low-resource languages. However, many previous works on cross-lingual transfer rely heavily on the parallel corpus or translation models, which are often difficult to obtain. We propose a novel approach to conduct zero-shot cross-lingual transfer with a pre-trained model. It consists of a Bilingual Task Fitting module that applies task-related bilingual information alignment; a self-training module generates pseudo soft and hard labels for unlabeled data and utilizes them to conduct self-training. We got the new SOTA on different tasks without any dependencies on the parallel corpus or translation models.

Introduction

In Natural Language Processing (NLP), high-resource languages like English benefit from vast amounts of labeled data. However, for the majority of the world's languages, labeled data is scarce. "Zero-shot cross-lingual transfer" aims to bridge this gap by training models on high-resource source languages and applying them to low-resource target languages. While multilingual models like mBERT provide a foundation, they often suffer from domain mismatch between general pre-training and specific downstream tasks. Existing solutions frequently rely on parallel corpora or machine translation, which are unavailable for many low-resource scenarios. This work introduces a framework that achieves cross-lingual transfer using only unlabeled target language data.

Methodology

The proposed framework consists of two primary stages: Bilingual Task Fitting (BTF) and Cross-lingual Self-training.

Overview of the proposed framework, including Bilingual Task Fitting and iterative Self-training.

Bilingual Task Fitting (BTF)

Before task-specific fine-tuning, the model undergoes a "warm-up" phase. It continues Masked Language Modeling (MLM) pre-training on a mixture of unlabeled source and target language data relevant to the task. To handle cases where task-related target data is missing, the researchers employ "entity matching": identifying entities in the source training set, locating their corresponding Wikipedia pages in the target language, and extracting those sentences to form a task-relevant synthetic corpus.

Cross-lingual Self-training

Following fine-tuning on the source language, the model enters an iterative self-training loop to adapt to the target language:

  1. Soft Labels: A "Teacher" model generates probability distributions for unlabeled target data, which a "Student" model mimics via Mean Squared Error (MSE) loss.
  2. Hard Labels & Voters Module: To mitigate noise, a "Voters Module"—an ensemble of multiple feature encoders—is used. Hard labels are assigned only when these voters agree with high confidence.
  3. Automated Thresholding: The confidence threshold ($\alpha$) is determined automatically by identifying the point of maximum curvature on the accuracy/recall curve of the labeled source language data, assuming the confidence-accuracy relationship is consistent across languages.
Accuracy/Recall curves for source and target languages used for automated thresholding.

Experimental Setup

The framework was evaluated on two benchmark tasks:

  • Cross-Lingual Sentiment (CLS): Sentiment classification using Amazon product reviews across English (source), German, French, and Japanese.
  • CoNLL (2002/2003): Named Entity Recognition (NER) across English, German, Spanish, and Dutch.

Results and Discussion

The method achieved new state-of-the-art (SOTA) results among non-parallel data methods. In sentiment analysis (CLS), the model reached an average accuracy of 85.21%, significantly exceeding the mBERT baseline of 76.49%. For NER (CoNLL), it achieved an average F1-score of 79.11%, outperforming other zero-shot methods that eschew translation.

Ablation studies highlighted the importance of the BTF stage; its removal led to a 7.55% accuracy drop on the CLS dataset, confirming that task-specific domain alignment is vital for effective transfer. Furthermore, the automated thresholding strategy proved robust, successfully using source language performance as a proxy for target language reliability.

Conclusion

This research demonstrates that high-quality cross-lingual transfer is possible without parallel corpora. By combining domain-specific alignment with a sophisticated self-training regime and automated thresholding, the framework significantly enhances the performance of multilingual models on low-resource tasks. Future work may explore tighter integration of the two stages and leveraging multiple source languages for even broader transfer capabilities.

Summary updated

Abstract

Recently, although pre-trained language models have achieved great success on multilingual NLP (Natural Language Processing) tasks, the lack of training data on many tasks in low-resource languages still limits their performance. One effective way of solving that problem is to transfer knowledge from rich-resource languages to low-resource languages. However, many previous works on cross-lingual transfer rely heavily on the parallel corpus or translation models, which are often difficult to obtain. We propose a novel approach to conduct zero-shot cross-lingual transfer with a pre-trained model. It consists of a Bilingual Task Fitting module that applies task-related bilingual information alignment; a self-training module generates pseudo soft and hard labels for unlabeled data and utilizes them to conduct self-training. We got the new SOTA on different tasks without any dependencies on the parallel corpus or translation models.
BibTeX
@misc{zhang2023zero,
  author = {Yuyang Zhang and Xiaofeng Han and Baojun Wang},
  title = {{Zero-shot Cross-lingual Transfer without Parallel Corpus}},
  year = {2023},
  eprint = {2310.04726},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.04726},
}
Figure