Introduction
Aligning word embedding spaces across different languages is a foundational task for unsupervised machine translation and cross-lingual information retrieval. Traditional methods for finding a mapping between languages like English and Italian often rely on parallel corpora or unstable adversarial training. This paper introduces a fully unsupervised approach using the Gromov-Wasserstein (GW) distance, which directly optimizes the relational geometry between languages to provide a robust, efficient, and theoretically grounded alignment solution.
The Challenge of Unaligned Spaces
Word embedding models like Word2Vec or FastText represent words as vectors in high-dimensional space. While words with similar meanings occupy similar relative positions within a single language's "cloud," these clouds have no shared coordinate system across languages.
Previous unsupervised attempts often utilized Generative Adversarial Networks (GANs) to find an initial alignment. However, GANs are notoriously difficult to tune, prone to mode collapse, and computationally expensive. Standard Optimal Transport (OT) frameworks also fail in this context because they require a predefined cost to move a point from one space to another—a cost that is unknown until the spaces are already aligned.
Method: Relational Alignment via Gromov-Wasserstein
The core innovation of this work is the application of the Gromov-Wasserstein distance. Unlike standard OT, which compares points directly, GW compares the relationships between points.
Instead of asking "How far is apple from manzana?", the GW objective asks: "Is the distance between apple and orange in English similar to the distance between manzana and naranja in Spanish?" By matching these internal structures, the algorithm can find an optimal coupling between vocabularies without any prior cross-lingual supervision.
To make this computationally feasible, the authors cast the alignment as a quadratic assignment problem and employ entropic regularization (the Sinkhorn-Knopp algorithm). This transforms a hard combinatorial problem into a smooth, differentiable one solvable via projected gradient descent. For massive vocabularies, they use a two-step strategy: first aligning the 20,000 most frequent words to create a "seed" mapping, then using a Procrustes projection to align the remaining words.
Key Results and Stability
The GW approach demonstrates several significant advantages over adversarial methods:
- State-of-the-Art Performance: On the MUSE benchmark, GW matches or exceeds previous best unsupervised methods, achieving ~81.7% P@1 for English-Spanish. On more difficult datasets with non-comparable corpora (Dinu et al.), GW significantly outperforms competitors.
- Monotonic Convergence: Unlike the oscillating nature of GAN training, the GW objective decreases monotonically. As shown in the training dynamics, the unsupervised GW objective closely tracks actual translation accuracy, making model selection straightforward.
- Efficiency: The method is remarkably fast, running in minutes on a standard quad-core CPU, whereas adversarial methods often require significant GPU resources.
Conclusion
By treating language alignment as a problem of matching relational geometries rather than point-to-point distances, the Gromov-Wasserstein framework provides a stable and efficient alternative to adversarial training. The method's ability to operate entirely on internal distances makes it a mathematically "correct" tool for word embeddings, which are inherently metric-based representations of meaning. Future work may explore stochastic versions of the algorithm to handle even larger vocabularies without the need for a two-step refinement process.
Summary updated