Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations

Mikel Artetxe, Gorka Labaka, Eneko Agirre

Introduction

Bilingual word embedding mappings are a cornerstone of cross-lingual natural language processing. Typically, embeddings are trained independently for two languages and then aligned into a shared vector space using a seed dictionary. Historically, this field has been fragmented, with various researchers advocating for different linear transformation techniques such as Ordinary Least Squares (OLS) regression, Canonical Correlation Analysis (CCA), or orthogonal transformations.

This paper, presented at AAAI-18, introduces a unifying multi-step framework that decomposes these disparate methods into a single underlying process. By identifying the optimal sequence of operations—including normalization, whitening, and re-weighting—the authors not only explain the relative success of previous approaches but also derive a new "best-of-all-worlds" mapping that establishes a new state-of-the-art in bilingual lexicon extraction.

A Unified Multi-Step Framework

The authors propose that almost all existing linear mapping methods can be viewed as specific configurations of a six-step pipeline:

  1. Step 0: Normalization. Length normalization and mean centering of the initial embeddings.
  2. Step 1: Whitening. Transforming embeddings so that all components have unit variance and are uncorrelated.
  3. Step 2: Orthogonal Mapping. Using Singular Value Decomposition (SVD) to find the transformation that maximizes cross-covariance between languages.
  4. Step 3: Re-weighting. Scaling components based on their cross-language correlation.
  5. Step 4: De-whitening. Restoring the original variance to the embeddings.
  6. Step 5: Dimensionality Reduction. Retaining only the most informative components.
Table 1: Mapping existing methods (OLS, CCA, Orthogonal) to the proposed multi-step framework (S0-S5).

As shown in the table above, traditional methods like OLS and CCA are essentially specific paths through this framework. For instance, OLS implicitly performs whitening, re-weighting, and de-whitening in a single, often suboptimal, step.

Key Insights: Directionality and Hubness

A critical discovery of this work is the importance of directionality in re-weighting. The authors found that re-weighting the target language embeddings is significantly more effective than re-weighting the source. This asymmetric approach directly addresses the "hubness problem"—a common failure mode where a few target words become the nearest neighbors for many source words, leading to poor translation accuracy. By re-weighting the target space, the model effectively alters the cosine similarity landscape to mitigate these hubs.

Furthermore, while whitening is beneficial for alignment, doing it "implicitly" (as in OLS) can introduce instability. The proposed method performs explicit whitening followed by careful re-weighting or de-whitening to maintain structural integrity.

Experimental Results

The framework was evaluated on bilingual lexicon extraction across four language pairs: English-Italian, English-German, English-Finnish, and English-Spanish. The "Proposed" configuration—combining whitening, orthogonal mapping, target-language re-weighting, and de-whitening—consistently outperformed all baselines.

Table 2: Comparison of the proposed method against various state-of-the-art baselines across four language pairs.

As seen in the results, the proposed method achieved 45.27% accuracy on the English-Italian benchmark using an inverted softmax retrieval method, a substantial improvement over the original OLS approach (34.93%) and the previous state-of-the-art (43.1%). The consistency of these gains across diverse language pairs like Finnish and Spanish underscores the robustness of the framework.

Conclusion

By unifying bilingual mapping techniques into a single linear framework, this research provides both theoretical clarity and a practical performance boost. While the current framework is strictly linear and requires a seed dictionary, its success suggests that careful management of embedding variance and retrieval metrics is key to cross-lingual alignment. Future work may explore reducing supervision or extending these insights to non-linear mapping strategies.

Summary updated

Abstract

Using a dictionary to map independently trained word embeddings to a shared space has shown to be an effective approach to learn bilingual word embeddings. In this work, we propose a multi-step framework of linear transformations that generalizes a substantial body of previous work. The core step of the framework is an orthogonal transformation, and existing methods can be explained in terms of the additional normalization, whitening, re-weighting, de-whitening and dimensionality reduction steps. This allows us to gain new insights into the behavior of existing methods, including the effectiveness of inverse regression, and design a novel variant that obtains the best published results in zero-shot bilingual lexicon extraction. The corresponding software is released as an open source project.
BibTeX
@inproceedings{artetxe2018generalizing,
  author = {Mikel Artetxe and Gorka Labaka and Eneko Agirre},
  title = {{Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11992},
}
Figure