TL;DR
This paper tackles inconsistencies in current bilingual word embedding methods by normalizing word vectors to a hypersphere and enforcing orthogonal transforms. This approach leads to improved performance on both word similarity tasks and English-to-Spanish translation, suggesting a more robust way to bridge language gaps in vector spaces.
What Problem Does This Paper Address?
Word embeddings have revolutionized how we represent words, capturing semantic relationships in a continuous vector space. A common approach to bilingual translation involves learning a linear transform to map word embeddings from one language to another. However, the authors identify a critical inconsistency: the objective functions used to learn word embeddings (often based on inner products) and the distance metrics used to evaluate them (commonly cosine distance) don't align. Furthermore, the linear transform used for bilingual mapping is often trained with a mean squared error objective, which is also at odds with the cosine distance used for evaluating translation quality. This mismatch can lead to suboptimal performance. The paper aims to resolve these inconsistencies by ensuring that both the embedding learning and the translation transform are aligned with the cosine distance metric.
Methods
The core of the paper's contribution lies in two key modifications to existing methods:
- Normalization of Word Vectors: Instead of allowing word vectors to exist anywhere in the embedding space, the authors enforce that all word vectors lie on a hypersphere (i.e., have a unit length). This is achieved by simply dividing each vector by its L2 norm whenever it's updated during training. This normalization has a crucial consequence: the inner product between two normalized vectors becomes equivalent to their cosine distance. This directly addresses the inconsistency between the embedding learning objective and the distance measurement.
- Orthogonal Transform for Bilingual Mapping: To maintain consistency with the normalized word vectors, the linear transform used for bilingual translation is constrained to be an orthogonal transform. An orthogonal matrix preserves lengths and angles, and when applied to normalized vectors, it also results in normalized vectors. This ensures that the projected vectors remain on the hypersphere. The optimization for this orthogonal transform is then performed using cosine distance as the objective. For efficiency, they use a gradient descent approach followed by an orthogonalization step (using SVD) to enforce the orthogonality constraint.
Datasets
- Monolingual Word Similarity: The WordSimilarity-353 Test Collection (Finkelstein et al., 2002) was used to evaluate the quality of monolingual word embeddings. This dataset consists of word pairs with human-judged similarity scores.
- Bilingual Word Translation: The authors used 6000 frequent English words and their translations to Spanish obtained via Google's translation service. These pairs were used for training and testing the bilingual transform using cross-validation. The data preparation involved tokenizing text from the EMNLP 2011 SMT workshop (WMT11) dataset.
Key Insights and Ideas
- The "Hypersphere" Insight: By normalizing word vectors to lie on a hypersphere, the paper elegantly bridges the gap between inner product-based learning and cosine distance-based evaluation. This is a fundamental insight that simplifies the problem.
- Orthogonality as a Natural Fit: Constraining the bilingual transform to be orthogonal is a logical consequence of normalizing word vectors. It ensures that the geometric relationships between words are preserved during translation, which is crucial for semantic accuracy.
- Inconsistency is the Enemy: The paper strongly argues that the inconsistencies in objective functions and distance metrics are not minor issues but significant contributors to suboptimal performance in bilingual translation.
Key Results
- Monolingual Word Similarity: Normalized word vectors showed a higher correlation with human judgments on the WordSimilarity-353 task compared to unnormalized vectors, indicating better quality embeddings.
- Bilingual Word Translation: The proposed method using normalized embeddings and an orthogonal transform consistently outperformed the baseline linear transform method. For instance, with specific dimension settings (800 for English, 200 for Spanish), the orthogonal transform achieved P@1 scores of 38.99% and P@5 scores of 59.16%, which are notably higher than the linear transform's best results (35.36% P@1 and 53.96% P@5).
Limitations and Future Work
- Dimension Mismatch: The paper acknowledges that handling cases where source and target language embedding dimensions differ is "rather ad-hoc." Their solution involves padding vectors with a tunable constant and re-normalizing, which they suggest could be improved.
- Task Specificity: The authors are unsure if their normalized embedding approach will generalize to other tasks like relation prediction, though they expect it might.
- Exploration of Hypersphere Properties: The paper suggests that placing word vectors on a hypersphere opens up new avenues for studying their properties in a less explored space.
Summary updated