Abstract: Xin Dong, Gerard de Melo. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Deep learning models are notoriously "data-hungry," requiring massive amounts of labeled text to perform well. While English-language datasets are abundant, many other languages suffer from a "low-resource" problem where labeled data is scarce or non-existent. Traditional solutions like human translation are prohibitively expensive, and machine translation often introduces noise or lacks domain-specific accuracy.
Cross-lingual transfer aims to bridge this gap by training a model in one language (usually English) and applying it to another. However, even powerful models like multilingual BERT (mBERT) often see a performance drop when moving from the source language to a distant target language. This paper introduces a simple yet powerful self-learning framework that boosts cross-lingual text classification by leveraging unlabeled data in target languages.
Methodology
The authors propose a Robust Self-Learning Framework consisting of three main components: a pretrained multilingual encoder (mBERT), a task-specific classification module, and a "balancing" selection mechanism.
The process follows an iterative loop:
- Initial Training: The model is first fine-tuned using available labeled English data.
- Prediction: The trained model makes predictions on a set of unlabeled non-English samples.
- Selection (The "Robust" Part): Instead of just taking the top $N$ most confident predictions, the authors use a balancing approach. They select an equal number of high-confidence samples for each target class. This prevents the model from becoming biased toward a majority class.
- Merging & Retraining: These selected "pseudo-labeled" samples are merged into the training set. The model is then retrained on this expanded, multilingual dataset.
- Iteration: This process repeats until a stopping criterion (like early stopping on a validation set) is met.
Experimental Setup
The framework was evaluated on two major tasks:
- MLDoc (Multilingual Document Classification): A balanced subset of the Reuters corpus covering 8 languages (English, German, Chinese, Spanish, French, Italian, Japanese, and Russian). It uses a 4-way topic classification (Corporate, Economics, Government, and Markets).
- Chinese Sentiment Classification: To test the model on a more distant language pair, the authors used 700,000 English Yelp reviews for training and evaluated the model on Chinese hotel reviews (150,000 unlabeled for self-learning, 10,000 for testing).
Results and Discussion
The self-learning approach outperformed vanilla mBERT and other strong baselines in 7 out of 8 languages on the MLDoc dataset. For example, on Russian (ru), accuracy jumped from 70.7% (vanilla BERT) to 79.3%.
On the English-to-Chinese sentiment task, the framework achieved 43.88% accuracy, outperforming the standard mBERT (40.73%) and adversarial methods like ADAN (42.49%).
Key Insights
- The Power of "Pseudo-Labels": The model is surprisingly good at labeling its own data. In the MLDoc experiments, the accuracy of the selected pseudo-labeled instances was between 80% and 93% across various languages.
- Balance is Critical: A key finding was that "throttling" (simply picking the most confident samples regardless of class) leads to rapid performance decline. This is because the model tends to get "stuck" in a feedback loop, over-representing the most frequent classes and amplifying its own errors. The "balancing" mechanism is what makes the framework robust.
- Bridging the Language Gap: The framework effectively "repurposes" the model's internal multilingual representations. By seeing actual target-language data during the fine-tuning phase, the model learns the specific nuances of how a task (like sentiment) is expressed in that language.
Conclusion
While the framework shows significant promise, it relies on the initial multilingual model being "good enough" to produce reasonably accurate first-pass predictions. Additionally, iterative retraining is more computationally expensive than a single fine-tuning pass. Future work could extend this framework to other NLP tasks beyond classification, such as sequence labeling or question answering.
Summary updated