Abstract: Pratik Jawanpuria, Mayank Meghwanshi, Bamdev Mishra. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Multilingual word embeddings (MWEs) are a cornerstone of cross-lingual Natural Language Processing, allowing models to represent words from different languages in a shared mathematical space. The "holy grail" of this field is unsupervised learning—aligning these spaces without the need for parallel text or human-translated dictionaries.
Existing state-of-the-art methods typically employ joint optimization, attempting to simultaneously find word alignments and learn the mathematical transformations to map them. However, this approach is notoriously unstable. When languages are linguistically distant (e.g., English vs. Arabic), these models often fail to converge, resulting in "collapsed" spaces where the embeddings lose their utility. This paper proposes a simpler, decoupled framework to address these stability issues.
Methodology
The authors introduce a multi-stage framework that breaks the unsupervised multilingual embedding problem into two distinct steps:
- Stage 1: Bilingual Lexicon Generation. Instead of aligning all languages at once, the framework uses existing unsupervised bilingual algorithms (such as self-learning or Gromov-Wasserstein optimal transport) to create dictionaries for a minimal set of language pairs (specifically $n-1$ pairs for $n$ languages).
- Stage 2: Multilingual Mapping. These dictionaries are then fed into the GeoMM (Geometry-aware Multilingual Mapping) algorithm. GeoMM learns language-specific rotations and a shared Mahalanobis metric, which governs how distances are measured in the final shared space.
The researchers evaluated two main variants: SL-GeoMM (using self-learning for Stage 1) and GW-GeoMM (using Gromov-Wasserstein).
Experimental Results
The framework was tested across standard benchmarks including MUSE, SemEval 2017, and ReutersMLDC, covering both closely related and linguistically diverse language sets.
Performance on Related Languages
On closely related European languages (English, French, German, etc.), the simple two-stage approach performed on par with complex joint models. As seen in the table below, SL-GeoMM achieved an average Precision@1 (P@1) of 79.0%, matching the performance of the UMWE baseline.
Robustness to Distant Languages
The primary strength of the proposed method is its robustness. While joint optimization methods often achieve 0% accuracy on distant language pairs due to convergence failure, the decoupled approach remains stable.
In a diverse setup involving six languages (including Arabic, Hindi, and Russian), SL-GeoMM achieved an average P@1 of 50.8%. In contrast, competing unsupervised multilingual methods like UMWE and UMH struggled significantly, with average scores dropping to 22.2% and 26.1%, respectively, due to frequent failures on specific language pairs.
Downstream Task Performance
The utility of these embeddings extends beyond lexicon induction. The authors demonstrated that their method leads to superior performance in practical applications:
- Multilingual Document Classification (MLDC): Achieved 90.3% accuracy.
- Multilingual Dependency Parsing (MLDP): Achieved 71.0% accuracy, outperforming several complex baselines.
Conclusion
By decoupling the alignment and mapping processes, this framework provides a more stable and flexible alternative to joint optimization. The modularity of the approach allows researchers to "plug and play" different alignment algorithms, ensuring high-quality multilingual spaces even for linguistically distant languages that previously posed significant challenges for unsupervised methods.
Summary updated