A Simple Approach to Learning Unsupervised Multilingual Embeddings

Pratik Jawanpuria, Mayank Meghwanshi, Bamdev Mishra

Abstract: Pratik Jawanpuria, Mayank Meghwanshi, Bamdev Mishra. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Multilingual word embeddings (MWEs) are a cornerstone of cross-lingual Natural Language Processing, allowing models to represent words from different languages in a shared mathematical space. The "holy grail" of this field is unsupervised learning—aligning these spaces without the need for parallel text or human-translated dictionaries.

Existing state-of-the-art methods typically employ joint optimization, attempting to simultaneously find word alignments and learn the mathematical transformations to map them. However, this approach is notoriously unstable. When languages are linguistically distant (e.g., English vs. Arabic), these models often fail to converge, resulting in "collapsed" spaces where the embeddings lose their utility. This paper proposes a simpler, decoupled framework to address these stability issues.

Methodology

The authors introduce a multi-stage framework that breaks the unsupervised multilingual embedding problem into two distinct steps:

  1. Stage 1: Bilingual Lexicon Generation. Instead of aligning all languages at once, the framework uses existing unsupervised bilingual algorithms (such as self-learning or Gromov-Wasserstein optimal transport) to create dictionaries for a minimal set of language pairs (specifically $n-1$ pairs for $n$ languages).
  2. Stage 2: Multilingual Mapping. These dictionaries are then fed into the GeoMM (Geometry-aware Multilingual Mapping) algorithm. GeoMM learns language-specific rotations and a shared Mahalanobis metric, which governs how distances are measured in the final shared space.

The researchers evaluated two main variants: SL-GeoMM (using self-learning for Stage 1) and GW-GeoMM (using Gromov-Wasserstein).

Experimental Results

The framework was tested across standard benchmarks including MUSE, SemEval 2017, and ReutersMLDC, covering both closely related and linguistically diverse language sets.

On closely related European languages (English, French, German, etc.), the simple two-stage approach performed on par with complex joint models. As seen in the table below, SL-GeoMM achieved an average Precision@1 (P@1) of 79.0%, matching the performance of the UMWE baseline.

P@1 performance on 6 European languages.

Robustness to Distant Languages

The primary strength of the proposed method is its robustness. While joint optimization methods often achieve 0% accuracy on distant language pairs due to convergence failure, the decoupled approach remains stable.

P@1 performance on a diverse set of 6 distant languages.

In a diverse setup involving six languages (including Arabic, Hindi, and Russian), SL-GeoMM achieved an average P@1 of 50.8%. In contrast, competing unsupervised multilingual methods like UMWE and UMH struggled significantly, with average scores dropping to 22.2% and 26.1%, respectively, due to frequent failures on specific language pairs.

Downstream Task Performance

The utility of these embeddings extends beyond lexicon induction. The authors demonstrated that their method leads to superior performance in practical applications:

  • Multilingual Document Classification (MLDC): Achieved 90.3% accuracy.
  • Multilingual Dependency Parsing (MLDP): Achieved 71.0% accuracy, outperforming several complex baselines.

Conclusion

By decoupling the alignment and mapping processes, this framework provides a more stable and flexible alternative to joint optimization. The modularity of the approach allows researchers to "plug and play" different alignment algorithms, ensuring high-quality multilingual spaces even for linguistically distant languages that previously posed significant challenges for unsupervised methods.

Summary updated

BibTeX
@inproceedings{jawanpuria2020simple,
  author = {Pratik Jawanpuria and Mayank Meghwanshi and Bamdev Mishra},
  title = {{A Simple Approach to Learning Unsupervised Multilingual Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=JXohXpzrK9},
}
Figure