A Robust Self-Learning Framework for Cross-Lingual Text Classification

Xin Dong, Gerard de MeloEMNLP 2019 · OpenReview

Abstract: Xin Dong, Gerard de Melo. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Deep learning models are notoriously "data-hungry," requiring massive amounts of labeled text to perform well. While English-language datasets are abundant, many other languages suffer from a "low-resource" problem where labeled data is scarce or non-existent. Traditional solutions like human translation are prohibitively expensive, and machine translation often introduces noise or lacks domain-specific accuracy.

Cross-lingual transfer aims to bridge this gap by training a model in one language (usually English) and applying it to another. However, even powerful models like multilingual BERT (mBERT) often see a performance drop when moving from the source language to a distant target language. This paper introduces a simple yet powerful self-learning framework that boosts cross-lingual text classification by leveraging unlabeled data in target languages.

Methodology

The authors propose a Robust Self-Learning Framework consisting of three main components: a pretrained multilingual encoder (mBERT), a task-specific classification module, and a "balancing" selection mechanism.

The proposed robust self-learning framework architecture.

The process follows an iterative loop:

  1. Initial Training: The model is first fine-tuned using available labeled English data.
  2. Prediction: The trained model makes predictions on a set of unlabeled non-English samples.
  3. Selection (The "Robust" Part): Instead of just taking the top $N$ most confident predictions, the authors use a balancing approach. They select an equal number of high-confidence samples for each target class. This prevents the model from becoming biased toward a majority class.
  4. Merging & Retraining: These selected "pseudo-labeled" samples are merged into the training set. The model is then retrained on this expanded, multilingual dataset.
  5. Iteration: This process repeats until a stopping criterion (like early stopping on a validation set) is met.

Experimental Setup

The framework was evaluated on two major tasks:

  • MLDoc (Multilingual Document Classification): A balanced subset of the Reuters corpus covering 8 languages (English, German, Chinese, Spanish, French, Italian, Japanese, and Russian). It uses a 4-way topic classification (Corporate, Economics, Government, and Markets).
  • Chinese Sentiment Classification: To test the model on a more distant language pair, the authors used 700,000 English Yelp reviews for training and evaluated the model on Chinese hotel reviews (150,000 unlabeled for self-learning, 10,000 for testing).

Results and Discussion

The self-learning approach outperformed vanilla mBERT and other strong baselines in 7 out of 8 languages on the MLDoc dataset. For example, on Russian (ru), accuracy jumped from 70.7% (vanilla BERT) to 79.3%.

Comparison of accuracy on the MLDoc dataset across eight languages.

On the English-to-Chinese sentiment task, the framework achieved 43.88% accuracy, outperforming the standard mBERT (40.73%) and adversarial methods like ADAN (42.49%).

Key Insights

  • The Power of "Pseudo-Labels": The model is surprisingly good at labeling its own data. In the MLDoc experiments, the accuracy of the selected pseudo-labeled instances was between 80% and 93% across various languages.
  • Balance is Critical: A key finding was that "throttling" (simply picking the most confident samples regardless of class) leads to rapid performance decline. This is because the model tends to get "stuck" in a feedback loop, over-representing the most frequent classes and amplifying its own errors. The "balancing" mechanism is what makes the framework robust.
  • Bridging the Language Gap: The framework effectively "repurposes" the model's internal multilingual representations. By seeing actual target-language data during the fine-tuning phase, the model learns the specific nuances of how a task (like sentiment) is expressed in that language.

Conclusion

While the framework shows significant promise, it relies on the initial multilingual model being "good enough" to produce reasonably accurate first-pass predictions. Additionally, iterative retraining is more computationally expensive than a single fine-tuning pass. Future work could extend this framework to other NLP tasks beyond classification, such as sequence labeling or question answering.

Summary updated

BibTeX
@inproceedings{dong2019robust,
  author = {Xin Dong and Gerard de Melo},
  title = {{A Robust Self-Learning Framework for Cross-Lingual Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=WdcJTlJSdTK},
}
Figure