Abstract
Traditional cross-lingual word embeddings rely on mapping static word vectors, which often struggle with words that have multiple meanings (polysemy). This paper introduces context-aware mapping, demonstrating that aligning contextualized embeddings (like ELMo) or mapping entire sentences directly leads to significantly more accurate cross-lingual representations, even for word-level tasks.
The Problem: The Limits of Static Alignment
To enable cross-lingual transfer in AI models, researchers typically align monolingual vector spaces into a shared "map." The standard approach involves learning a linear transformation (often an orthogonal rotation) that maps a bilingual dictionary from one space to another.
However, this approach relies on static word vectors (e.g., FastText), where a word like "bank" is represented by a single point regardless of whether it refers to a financial institution or a river bank. If the alignment dictionary maps "bank" to a financial term in Spanish, but the training data uses it in a geographic context, the resulting mapping becomes skewed. This "one-size-fits-all" approach creates noise that limits performance in downstream tasks like sentence translation.
Proposed Methods: Incorporating Context
The authors propose two primary methods to integrate context into the alignment process:
- Mapping Contextualized Embeddings: Instead of static vectors, the authors utilize ELMo. Because ELMo produces different vectors for the same word depending on its context, they use a parallel corpus with word alignments to extract a "dynamic dictionary." For instance, they extract the specific ELMo vector for "bank" when it is aligned with the Spanish "orilla" (river bank) to ensure the mapping is sense-specific.
- Sentence-Level Mapping: Rather than aligning individual words, they align the average vector of entire sentences. The intuition is that a sentence is far less ambiguous than a standalone word. By learning a transformation matrix that maps the representation of an English sentence to its Spanish translation, the model captures a robust global alignment less sensitive to individual word misalignments.
Both methods utilize the Orthogonal Procrustes problem to find the best rotation to align the two sets of points while preserving the internal geometric distances between words in each language.
Key Results and Insights
The researchers evaluated their methods on sentence translation retrieval and word-level tasks across English, Spanish, and German.
1. Context as a Disambiguator
By using contextualized embeddings, the authors effectively perform implicit word sense disambiguation. This prevents the mapping from being "pulled" in the wrong direction by homographs. ELMo-based word alignment achieved the highest accuracies across the board, reaching ~84% accuracy in English → Spanish retrieval compared to ~74% for standard FastText word mapping.
2. Sentence-Level Generalization
A significant finding was that a transformation matrix learned entirely on sentence averages works just as well—and sometimes better—for word-level translation. For FastText, sentence-level mapping yielded a ~3% improvement over word-level mapping across all language directions. This suggests that sentence-level alignment captures the underlying geometric relationship between languages more cleanly than word-level dictionaries.
3. Data Efficiency
ELMo-based word alignment proved remarkably data-efficient. It outperformed the static dictionary baseline using as few as 2,000 parallel sentences, making it a viable option for lower-resource scenarios where massive parallel corpora are unavailable.
Conclusion
Context-aware mapping addresses the fundamental limitations of static word vector alignment. By leveraging contextualized embeddings or sentence-level averages, researchers can create more robust and accurate cross-lingual shared spaces. While these methods still require some parallel data, their efficiency and performance gains—particularly in zero-shot scenarios—mark a significant step forward for cross-lingual NLP.
Summary updated