A Call for More Rigor in Unsupervised Cross-lingual Learning

Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, Eneko Agirre

Introduction

Unsupervised Cross-lingual Learning (UCL) has emerged as a prominent research area in Natural Language Processing, promising the ability to bridge the digital divide for the world's 6,000+ languages without the need for parallel data. However, this position paper argues that the standard justification for UCL—the "zero parallel data" scenario—is largely a myth. In practice, languages with sufficient monolingual data for unsupervised training almost always possess some form of parallel supervision, such as religious texts or shared entities. This work calls for a shift toward scientific rigor and transparent reporting of the "hidden" signals that drive these models.

Categorizing Cross-lingual Signals

UCL models, including Cross-lingual Word Embeddings (CLWE), deep multilingual pretraining (e.g., mBERT, XLM), and Unsupervised Machine Translation (UNMT), often rely on implicit signals rather than pure unsupervised induction. These signals can be categorized into monolingual properties and cross-lingual links.

Taxonomy of signals used in unsupervised cross-lingual learning.

As shown in the table above, signals like shared writing systems, identical words (numerals, URLs), and string similarity provide a form of "hidden supervision." The authors argue that these should be explicitly acknowledged as cross-lingual signals rather than being treated as emergent properties of unsupervised learning.

Methodological Issues and Evaluation

A critical critique of the field reveals several methodological pitfalls. One of the most significant is the "validation leak," where researchers use parallel validation sets to tune hyperparameters for supposedly unsupervised models. This practice effectively renders the models semi-supervised.

Summary of common methodological issues in UCL research.

Furthermore, evaluation often occurs under favorable conditions, such as between typologically similar languages or within the same domain. Current benchmarks like XNLI and MLDoc, often created via translation, may introduce artificial artifacts that do not reflect real-world low-resource scenarios. The lack of established benchmarks often leads to inconsistent reporting and overfitting to specific tasks like bilingual lexicon induction.

Recommendations for Future Research

To improve the scientific standing of the field, the authors propose four key recommendations:

  1. Rigorous Motivation: Avoid claiming UCL is necessary for "no parallel data" scenarios unless specifically proven for the target language. Often, resources like the Multilingual Bible or JW300 provide parallel data for hundreds of languages.
  2. Explicit Signal Reporting: Clearly state dependencies on shared vocabularies, subwords, or linguistic similarities.
  3. Transparent Validation: Minimize and report any use of parallel data for hyperparameter tuning. There is a pressing need for better unsupervised validation criteria.
  4. Holistic Evaluation: Utilize diverse, non-English-centric tasks and account for linguistic distance between the languages involved.

Conclusion

While UCL remains a fascinating scientific frontier for testing the distributional hypothesis—the idea that words in similar contexts have similar meanings—its current practical utility for low-resource languages is often overstated. Future work should explore "tabula rasa" approaches that do not assume shared scripts or word segmentation to truly test the limits of machine translation between alien systems.

Summary updated

Abstract

We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of parallel data for many of the world's languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting. We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.
BibTeX
@inproceedings{artetxe2020call,
  author = {Mikel Artetxe and Sebastian Ruder and Dani Yogatama and Gorka Labaka and Eneko Agirre},
  title = {{A Call for More Rigor in Unsupervised Cross-lingual Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.658},
}
Figure