Introduction
Bilingual word embedding mappings are a cornerstone of cross-lingual natural language processing. Typically, embeddings are trained independently for two languages and then aligned into a shared vector space using a seed dictionary. Historically, this field has been fragmented, with various researchers advocating for different linear transformation techniques such as Ordinary Least Squares (OLS) regression, Canonical Correlation Analysis (CCA), or orthogonal transformations.
This paper, presented at AAAI-18, introduces a unifying multi-step framework that decomposes these disparate methods into a single underlying process. By identifying the optimal sequence of operations—including normalization, whitening, and re-weighting—the authors not only explain the relative success of previous approaches but also derive a new "best-of-all-worlds" mapping that establishes a new state-of-the-art in bilingual lexicon extraction.
A Unified Multi-Step Framework
The authors propose that almost all existing linear mapping methods can be viewed as specific configurations of a six-step pipeline:
- Step 0: Normalization. Length normalization and mean centering of the initial embeddings.
- Step 1: Whitening. Transforming embeddings so that all components have unit variance and are uncorrelated.
- Step 2: Orthogonal Mapping. Using Singular Value Decomposition (SVD) to find the transformation that maximizes cross-covariance between languages.
- Step 3: Re-weighting. Scaling components based on their cross-language correlation.
- Step 4: De-whitening. Restoring the original variance to the embeddings.
- Step 5: Dimensionality Reduction. Retaining only the most informative components.
As shown in the table above, traditional methods like OLS and CCA are essentially specific paths through this framework. For instance, OLS implicitly performs whitening, re-weighting, and de-whitening in a single, often suboptimal, step.
Key Insights: Directionality and Hubness
A critical discovery of this work is the importance of directionality in re-weighting. The authors found that re-weighting the target language embeddings is significantly more effective than re-weighting the source. This asymmetric approach directly addresses the "hubness problem"—a common failure mode where a few target words become the nearest neighbors for many source words, leading to poor translation accuracy. By re-weighting the target space, the model effectively alters the cosine similarity landscape to mitigate these hubs.
Furthermore, while whitening is beneficial for alignment, doing it "implicitly" (as in OLS) can introduce instability. The proposed method performs explicit whitening followed by careful re-weighting or de-whitening to maintain structural integrity.
Experimental Results
The framework was evaluated on bilingual lexicon extraction across four language pairs: English-Italian, English-German, English-Finnish, and English-Spanish. The "Proposed" configuration—combining whitening, orthogonal mapping, target-language re-weighting, and de-whitening—consistently outperformed all baselines.
As seen in the results, the proposed method achieved 45.27% accuracy on the English-Italian benchmark using an inverted softmax retrieval method, a substantial improvement over the original OLS approach (34.93%) and the previous state-of-the-art (43.1%). The consistency of these gains across diverse language pairs like Finnish and Spanish underscores the robustness of the framework.
Conclusion
By unifying bilingual mapping techniques into a single linear framework, this research provides both theoretical clarity and a practical performance boost. While the current framework is strictly linear and requires a seed dictionary, its success suggests that careful management of embedding variance and retrieval metrics is key to cross-lingual alignment. Future work may explore reducing supervision or extending these insights to non-linear mapping strategies.
Summary updated