Abstract
Standard cross-lingual word embeddings (CLWE) often "underfit" their training dictionaries to achieve better generalization on word-translation tests. This paper demonstrates that by intentionally overfitting the training dictionary through a post-processing step called "retrofitting," researchers can significantly improve performance on real-world downstream tasks like document classification and dependency parsing, despite a decrease in traditional translation metrics.
Introduction
Cross-lingual word embeddings (CLWE) are fundamental to multilingual NLP, enabling models to share knowledge across languages by mapping words into a single shared vector space. Traditionally, these models are evaluated using Bilingual Lexicon Induction (BLI)—a translation test where the model identifies the correct target language word for a given source word.
To perform well on BLI tests, researchers typically employ linear projections. While these projections are smooth enough to generalize to unseen test words, they often fail to perfectly align the words in the training dictionary. This phenomenon, known as underfitting, occurs because the linear model prioritizes a general transformation over specific training pairs. The authors argue that BLI is a flawed primary metric because training dictionaries usually contain high-frequency, highly predictive words. Underfitting these common words to score higher on rare test words can degrade performance on complex downstream tasks that rely on frequent vocabulary.
The Retrofitting Framework
The authors propose a simple, two-stage pipeline to address this underfitting:
- Standard Projection: First, a standard CLWE is trained using a linear projection method such as Procrustes analysis (PROC), Canonical Correlation Analysis (CCA), or Relaxed Cross-Domain Similarity Local Scaling (RCSLS).
- Retrofitting: A post-processing step is applied to "pull" the training translation pairs closer together in the embedding space.
This method utilizes the retrofitting algorithm proposed by Faruqui et al. (2015), which balances two goals: maintaining the proximity of new embeddings to the originals while ensuring that translation pairs in a dictionary are as close as possible. The paper explores two types of dictionaries: the Training Dictionary (gold-standard pairs) and a Synthetic Dictionary (induced from the original CLWE using retrieval heuristics) to preserve alignments for words outside the training set.
The BLI Paradox and Overfitting as a Feature
In most machine learning contexts, overfitting is considered a failure. However, in CLWE, the authors demonstrate that overfitting the training dictionary is actually a feature. Because training dictionaries often contain the most frequent and "useful" words (e.g., "good" vs. the test set's "torpedo"), perfectly aligning them is critical for real-world utility.
As shown in the experimental results, retrofitting to the training dictionary resulted in nearly 100% BLI accuracy on the training set across all languages. While this caused BLI test scores to drop—sometimes by over 10 points—it significantly improved the model's utility for downstream applications.
Key Results and Downstream Impact
Despite the drop in BLI test scores, retrofitting improved average accuracy on both MLDoc (multilingual document classification) and Universal Dependencies (dependency parsing) across almost all tested languages and projection methods.
The study found that retrofitting to a combination of training and synthetic dictionaries often yielded the "best of both worlds," striking a balance between fitting high-utility training words and maintaining the generalization capabilities of the original linear projection.
Conclusion
The findings suggest that the NLP community should reconsider its reliance on BLI test accuracy as the gold standard for CLWE evaluation. The importance of specific words varies by task; for instance, function words are vital for dependency parsing but less so for classification. Future work should move toward evaluating embeddings directly on downstream metrics and exploring non-linear projections that can fit dictionaries more expressively from the start.
Summary updated