Abstract
Recent progress on unsupervised learning of cross-lingual embeddings in bilingual settings has given impetus to learning a shared embedding space for several languages without any supervision. A popular framework to solve the latter problem is to jointly solve the following two sub-problems: 1) learning unsupervised word alignment between several pairs of languages, and 2) learning how to map the monolingual embeddings of every language to a shared multilingual space. In contrast, we propose a simple, two-stage framework in which we decouple the above two sub-problems and solve them separately using existing techniques. The proposed approach obtains surprisingly good performance in various tasks such as bilingual lexicon induction, cross-lingual word similarity, multilingual document classification, and multilingual dependency parsing. When distant languages are involved, the proposed solution illustrates robustness and outperforms existing unsupervised multilingual word embedding approaches. Overall, our experimental results encourage development of multi-stage models for such challenging problems.
Introduction: The Challenge of Joint Optimization
In Natural Language Processing (NLP), Multilingual Word Embeddings (MWEs) serve as a foundational tool, mapping words from various languages into a single shared vector space. Ideally, semantically similar words across different languages—like "apple" (English), "pomme" (French), and "seb" (Hindi)—should be positioned close to each other in this space.
The prevailing trend in the field has been to learn these spaces in an unsupervised manner by jointly optimizing two sub-problems: alignment (identifying word correspondences) and mapping (mathematically rotating language "clouds" into a shared space). However, this joint approach suffers from inherent instability. If the model fails to align a single difficult language pair (e.g., English and Arabic), this failure can "poison" the entire shared space, degrading performance even for closely related languages.
The UMML Framework: Decoupling for Robustness
To address these instabilities, we propose UMML (Unsupervised Multilingual Multi-stage Learning). This framework decouples the alignment and mapping phases into two sequential stages:
- Stage 1: Independent Bilingual Alignment: We utilize existing unsupervised bilingual tools, such as Gromov-Wasserstein (GW) optimal transport or self-learning (SL) methods like VecMap, to create seed dictionaries for specific language pairs. These are learned independently and in parallel.
- Stage 2: Multilingual Mapping: These dictionaries are then fed into GeoMM (Geometry-aware Multilingual Mapping). GeoMM treats languages as nodes in a graph, learning a shared metric space and language-specific rotations to unify them.
By modularizing the process, a failure in one difficult alignment does not compromise the mathematical integrity of the rest of the multilingual space.
Experimental Evaluation
We evaluated UMML across several benchmarks, including MUSE for Bilingual Lexicon Induction (BLI), SemEval 2017 for word similarity, and ReutersMLDC for document classification.
Bilingual Lexicon Induction (BLI)
On a benchmark of 11 European languages, UMML consistently outperformed joint models. As shown in the results below, the self-learning variant (UMML-SL) achieved an average Precision@1 of 66.6%, a significant improvement over the joint UMWE model's 54.0%.
Robustness with Distant Languages
The most significant advantage of the two-stage approach is its robustness when dealing with structurally diverse languages like Arabic and Hindi. Joint models often collapse in these scenarios, failing to find any meaningful alignment. In a setup involving Arabic, German, English, French, Hindi, and Russian, UMML-SL maintained a stable average precision of 50.8%, whereas joint models like UMH and UMWE dropped to 26.1% and 22.2%, respectively.
Key Insights
- Simplicity and Modularity: "End-to-end" is not always optimal. A modular approach allows researchers to swap in the best-available bilingual tools for Stage 1 without redesigning the entire architecture.
- Knowledge Transfer: Although dictionaries are learned independently, the second stage (GeoMM) facilitates knowledge transfer across the shared space, improving generalization.
- Downstream Performance: UMML-SL also achieved state-of-the-art results in downstream tasks, such as multilingual document classification (90.3% accuracy).
Conclusion
Our findings suggest that decoupling complex unsupervised problems into manageable stages can lead to more robust and higher-performing models. UMML provides a simple yet powerful framework for learning multilingual embeddings, particularly when bridging the gap between distant languages.
Summary updated