Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring

Aitor Ormazabal, Mikel Artetxe, Aitor Soroa, Gorka Labaka, Eneko Agirre

Abstract

Traditional cross-lingual word embedding (CLWE) methods rely on offline mapping, which assumes that independently trained monolingual embedding spaces share a similar geometric structure (isometry). However, this assumption often fails in practice, leading to poor alignment. We present Context Anchoring, a novel approach that fixes a target language's embedding space and trains the source language from scratch using translated context words as anchors. This method effectively bypasses structural mismatches and achieves state-of-the-art results on bilingual lexicon induction, even when initialized with minimal seeds like numerals.

Introduction

Cross-lingual word embeddings (CLWEs) are essential for cross-lingual transfer in natural language processing. For years, the dominant paradigm has been offline mapping: training monolingual models separately and then finding a linear transformation to align them. This approach is predicated on the isometry assumption—the belief that word vector distributions across languages are roughly isomorphic. Recent research suggests this assumption is frequently violated due to differences in training corpora and language-specific nuances. When the underlying structures diverge, linear mapping cannot achieve perfect alignment.

Methods: From Mapping to Anchoring

Instead of attempting to align two pre-existing structures, we propose building the source language space around a fixed target language space.

Context Anchoring

We modify the Skip-gram with Negative Sampling (SGNS) algorithm. During training on source language text (e.g., Spanish), if a context word appears in a bilingual dictionary, the model uses the corresponding fixed vector from the target language (e.g., English) as an "anchor." This forces the source word vectors into the same geometric neighborhood as their target counterparts from the outset.

Self-Learning and Iterative Restarts

To minimize reliance on large human-made dictionaries, the system employs an iterative self-learning process. It can begin with a "weak" seed, such as identical strings or even just numerals. As training progresses, the model uses its current embeddings to induce a better dictionary via Cross-Domain Similarity Local Scaling (CSLS) retrieval. To avoid local optima, the process uses Iterative Restarts, where the dictionary is updated and the source embeddings are re-initialized and re-trained.

Performance (P@1) over re-induction iterations for different initialization strategies.

As shown in the figure above, even the "numeral init" strategy (starting only with numbers) eventually converges to high performance, demonstrating the robustness of the iterative anchoring approach compared to traditional mapping baselines like VecMap.

Key Results

Our method was evaluated on the MUSE dataset for Bilingual Lexicon Induction (BLI) across seven languages.

Superior Accuracy

Context Anchoring consistently outperformed established baselines. On the MUSE benchmark, our method achieved an average Precision@1 (P@1) of 67.5%, surpassing VecMap (65.1%) and significantly outperforming the original MUSE mapping (45.8%). The gains were particularly notable in "backward" induction (translating from the target English back to the source language).

Bilingual Lexicon Induction (BLI) results (P@1) comparing the proposed context anchoring method against various baselines.

Downstream Performance and Robustness

Beyond word-to-word translation, the embeddings were tested on the XNLI (Cross-lingual Natural Language Inference) task. The competitive results on XNLI indicate that context anchoring preserves deep semantic meaning suitable for complex downstream tasks. Furthermore, error analysis showed that our method is more robust at handling named entities, which mapping methods often struggle to translate correctly.

Conclusion

Context Anchoring represents a shift away from the flawed isometry assumption of offline mapping. By forcing the source language to adopt the target's geometric structure during training, we eliminate the primary cause of alignment failure. The ability to bootstrap from minimal seeds like numerals makes this approach particularly promising for low-resource languages where large bilingual dictionaries are unavailable. Future work will explore adversarial techniques to remove the need for seed dictionaries entirely.

Summary updated

BibTeX
@inproceedings{ormazabal2021beyond,
  author = {Aitor Ormazabal and Mikel Artetxe and Aitor Soroa and Gorka Labaka and Eneko Agirre},
  title = {{Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.506},
}
Figure