Abstract
Recently, although pre-trained language models have achieved great success on multilingual NLP (Natural Language Processing) tasks, the lack of training data on many tasks in low-resource languages still limits their performance. One effective way of solving that problem is to transfer knowledge from rich-resource languages to low-resource languages. However, many previous works on cross-lingual transfer rely heavily on the parallel corpus or translation models, which are often difficult to obtain. We propose a novel approach to conduct zero-shot cross-lingual transfer with a pre-trained model. It consists of a Bilingual Task Fitting module that applies task-related bilingual information alignment; a self-training module generates pseudo soft and hard labels for unlabeled data and utilizes them to conduct self-training. We got the new SOTA on different tasks without any dependencies on the parallel corpus or translation models.
Introduction
In Natural Language Processing (NLP), high-resource languages like English benefit from vast amounts of labeled data. However, for the majority of the world's languages, labeled data is scarce. "Zero-shot cross-lingual transfer" aims to bridge this gap by training models on high-resource source languages and applying them to low-resource target languages. While multilingual models like mBERT provide a foundation, they often suffer from domain mismatch between general pre-training and specific downstream tasks. Existing solutions frequently rely on parallel corpora or machine translation, which are unavailable for many low-resource scenarios. This work introduces a framework that achieves cross-lingual transfer using only unlabeled target language data.
Methodology
The proposed framework consists of two primary stages: Bilingual Task Fitting (BTF) and Cross-lingual Self-training.
Bilingual Task Fitting (BTF)
Before task-specific fine-tuning, the model undergoes a "warm-up" phase. It continues Masked Language Modeling (MLM) pre-training on a mixture of unlabeled source and target language data relevant to the task. To handle cases where task-related target data is missing, the researchers employ "entity matching": identifying entities in the source training set, locating their corresponding Wikipedia pages in the target language, and extracting those sentences to form a task-relevant synthetic corpus.
Cross-lingual Self-training
Following fine-tuning on the source language, the model enters an iterative self-training loop to adapt to the target language:
- Soft Labels: A "Teacher" model generates probability distributions for unlabeled target data, which a "Student" model mimics via Mean Squared Error (MSE) loss.
- Hard Labels & Voters Module: To mitigate noise, a "Voters Module"—an ensemble of multiple feature encoders—is used. Hard labels are assigned only when these voters agree with high confidence.
- Automated Thresholding: The confidence threshold ($\alpha$) is determined automatically by identifying the point of maximum curvature on the accuracy/recall curve of the labeled source language data, assuming the confidence-accuracy relationship is consistent across languages.
Experimental Setup
The framework was evaluated on two benchmark tasks:
- Cross-Lingual Sentiment (CLS): Sentiment classification using Amazon product reviews across English (source), German, French, and Japanese.
- CoNLL (2002/2003): Named Entity Recognition (NER) across English, German, Spanish, and Dutch.
Results and Discussion
The method achieved new state-of-the-art (SOTA) results among non-parallel data methods. In sentiment analysis (CLS), the model reached an average accuracy of 85.21%, significantly exceeding the mBERT baseline of 76.49%. For NER (CoNLL), it achieved an average F1-score of 79.11%, outperforming other zero-shot methods that eschew translation.
Ablation studies highlighted the importance of the BTF stage; its removal led to a 7.55% accuracy drop on the CLS dataset, confirming that task-specific domain alignment is vital for effective transfer. Furthermore, the automated thresholding strategy proved robust, successfully using source language performance as a proxy for target language reliability.
Conclusion
This research demonstrates that high-quality cross-lingual transfer is possible without parallel corpora. By combining domain-specific alignment with a sophisticated self-training regime and automated thresholding, the framework significantly enhances the performance of multilingual models on low-resource tasks. Future work may explore tighter integration of the two stages and leveraging multiple source languages for even broader transfer capabilities.
Summary updated