Context-Aware Cross-Lingual Mapping

Hanan Aldarmaki, Mona T. Diab

Abstract: Cross-lingual word vectors are typically obtained by fitting an orthogonal matrix that maps the entries of a bilingual dictionary from a source to a target vector space. Word vectors, however, are most commonly used for sentence or document-level representations that are calculated as the weighted average of word embeddings. In this paper, we propose an alternative to word-level mapping that better reflects sentence-level cross-lingual similarity. We incorporate context in the transformation matrix by directly mapping the averaged embeddings of aligned sentences in a parallel corpus. We also implement cross-lingual mapping of deep contextualized word embeddings using parallel sentences with word alignments. In our experiments, both approaches resulted in cross-lingual sentence embeddings that outperformed context-independent word mapping in sentence translation retrieval. Furthermore, the sentence-level transformation could be used for word-level mapping without loss in word translation quality.

Introduction

The standard approach to creating cross-lingual word embeddings involves training monolingual models independently and then learning a linear transformation to align them. This alignment is typically achieved by finding an orthogonal matrix that maps words from a source language dictionary to their counterparts in a target language. While effective, this method suffers from a fundamental limitation: lexical ambiguity.

Static word embeddings assign a single vector to each word, regardless of its usage. For example, the English word "bank" is mapped to a single point in space, conflating its financial and geographical meanings. When aligning this point to another language where these senses are distinct words, noise is inevitably introduced. This paper explores how incorporating sentential context can resolve such ambiguities and lead to more robust cross-lingual representations.

Methodology

The authors propose two primary strategies to move beyond context-independent mapping:

1. Mapping Contextualized Embeddings

Instead of static vectors like Word2Vec or FastText, this method utilizes ELMo (Embeddings from Language Models). ELMo generates dynamic vectors for words based on their surrounding context. Since traditional dictionaries cannot be used with dynamic vectors, the authors extract a "dynamic dictionary" from a parallel corpus. By using word alignments (e.g., IBM Model 2), they identify pairs of contextualized word vectors that correspond to each other across languages and learn a transformation matrix from these pairs.

2. Sentence-Level Mapping

This approach shifts the focus from individual words to entire sentences. By aligning the average vector of aligned sentence pairs from a parallel corpus, the model leverages the fact that full sentences are significantly less ambiguous than standalone words. A single transformation matrix is learned to minimize the distance between these aligned sentence representations.

Both methods employ the Orthogonal Procrustes approach, which ensures that the internal geometric structure of the original monolingual vector spaces is preserved during the alignment process.

Experimental Setup

  • Monolingual Training: Models were trained on the 1 Billion Word benchmark for English and WMT’13 news crawl data for Spanish and German.
  • Alignment Data: Subsets of parallel sentences from WMT’13, ranging from 100 to 1,000,000 sentences, were used to learn the mapping.
  • Evaluation Tasks:
    • Sentence Translation Retrieval: Identifying the correct translation of a source sentence from a target language pool.
  • Word-Level Evaluation: Assessing word translation precision and performance on the SemEval’17 cross-lingual word similarity task.

Key Results

The experimental results demonstrate that context-aware mapping significantly improves cross-lingual alignment, particularly for sentence-level tasks.

Nearest Neighbor (NN) accuracy for sentence translation retrieval across various language pairs and training data sizes.

Sentence Retrieval Performance

As shown in the figure above, ELMo-based word alignment consistently achieved the highest accuracies. For static FastText embeddings, switching from traditional word-level mapping to sentence-level mapping resulted in a consistent performance gain (approximately 3% on average). Interestingly, sentence-level mapping begins to reliably outperform word-level mapping once the alignment corpus reaches a threshold of roughly 50,000 sentences.

Data Efficiency and Zero-Shot Capabilities

One of the most striking findings is the data efficiency of the ELMo-based approach. It outperformed the static dictionary baseline with as few as 2,000 parallel sentences, making it highly suitable for low-resource language pairs. Furthermore, the models exhibited strong "zero-shot" performance. For instance, a model trained to map Spanish to English and German to English could successfully retrieve German translations for Spanish sentences without ever seeing Spanish-German parallel data.

Cross-Task Generalization

A surprising insight was that the transformation matrix learned at the sentence level performed as well as, or better than, word-level matrices on word-translation tasks. This suggests that the sentence-level signal provides a cleaner, more representative alignment of the underlying semantic spaces than noisy word-level dictionaries.

Conclusion

By incorporating context into the cross-lingual mapping process, this work addresses a major bottleneck in traditional alignment methods. Whether through deep contextualized embeddings or direct sentence-level alignment, context-aware mapping provides superior performance in sentence retrieval and maintains high quality in word-level tasks. Future work may extend these techniques to more recent architectures like BERT or other transformer-based encoders to further enhance cross-lingual transfer learning.

Summary updated

Abstract

Cross-lingual word vectors are typically obtained by fitting an orthogonal matrix that maps the entries of a bilingual dictionary from a source to a target vector space. Word vectors, however, are most commonly used for sentence or document-level representations that are calculated as the weighted average of word embeddings. In this paper, we propose an alternative to word-level mapping that better reflects sentence-level cross-lingual similarity. We incorporate context in the transformation matrix by directly mapping the averaged embeddings of aligned sentences in a parallel corpus. We also implement cross-lingual mapping of deep contextualized word embeddings using parallel sentences with word alignments. In our experiments, both approaches resulted in cross-lingual sentence embeddings that outperformed context-independent word mapping in sentence translation retrieval. Furthermore, the sentence-level transformation could be used for word-level mapping without loss in word translation quality.
BibTeX
@inproceedings{aldarmaki2019context,
  author = {Hanan Aldarmaki and Mona T. Diab},
  title = {{Context-Aware Cross-Lingual Mapping}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=SkZpP7W_ZH},
}
Figure