A Call for More Rigor in Unsupervised Cross-lingual Learning

Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, Eneko Agirre

Abstract

We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of parallel data for many of the world's languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting. We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.

Introduction

Unsupervised Cross-lingual Learning (UCL) has emerged as a prominent research area, aiming to bridge the gap between languages using only raw, monolingual text. The standard motivation for this pursuit is the scarcity of parallel data for the majority of the world's languages. However, this paper argues that the "no parallel data but abundant monolingual data" scenario is largely a myth. In practice, languages with sufficient digital presence to support massive unsupervised training (like those in Wikipedia or Common Crawl) almost always possess some parallel corpora (e.g., religious texts or mined bitexts). Conversely, truly low-resource languages lack both parallel and monolingual data. Consequently, UCL should be viewed more as a scientific endeavor to understand language universals rather than a purely practical solution for low-resource scenarios.

Training Signals in UCL

A critical aspect of UCL research is the definition of "unsupervised." Many models rely on implicit cross-lingual signals that provide a form of hidden supervision. These signals can be categorized into monolingual features and cross-lingual bridges.

Summary of training signals used in unsupervised cross-lingual learning.

As shown in the table above, cross-lingual signals often include shared writing systems or identical words (anchors). Relying on these signals means that a model's success may depend more on the typological similarity of the languages (e.g., English and French) than on the robustness of the unsupervised algorithm itself. Researchers must be transparent about these assumptions to ensure fair comparisons across different language pairs.

Methodological Challenges

The field currently suffers from several methodological inconsistencies, particularly regarding how models are tuned and evaluated. A common pitfall is "validation cheating," where researchers use parallel validation sets to tune hyperparameters while claiming the model is unsupervised.

Common methodological issues and illustrative examples in UCL research.

Beyond validation, evaluation often occurs under "favorable conditions." Models are frequently tested on closely related European languages or within the same domain (e.g., news), which does not reflect real-world cross-lingual generalization. There is also an over-reliance on specific tasks like Bilingual Lexicon Induction (BLI), which may not translate to performance on complex downstream tasks.

Evaluation and Benchmarking

The paper critiques the use of older benchmarks like MLDoc, which can often be solved via simple keyword matching, and XNLI, which may contain translation artifacts. Instead, the authors advocate for modern, diverse, and multi-task benchmarks such as XTREME, XGLUE, and TyDi QA. These benchmarks provide a more rigorous testing ground for cross-lingual models across a wider variety of language families and tasks.

Conclusion

To advance the field of unsupervised cross-lingual learning, the community must move beyond the "low-resource" narrative and embrace UCL as a scientific challenge. This requires a paradigm shift toward more rigorous validation practices, transparency regarding training signals, and evaluation on diverse language typologies. By bridging the silos between cross-lingual embeddings, deep pretraining, and machine translation, we can develop a more unified and robust understanding of how models learn across languages.

Summary updated

Abstract

We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of parallel data for many of the world's languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting. We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.
BibTeX
@inproceedings{artetxe2020call,
  author = {Mikel Artetxe and Sebastian Ruder and Dani Yogatama and Gorka Labaka and Eneko Agirre},
  title = {{A Call for More Rigor in Unsupervised Cross-lingual Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.658},
}
Figure