Abstract
Cross-lingual word embeddings (CLWEs) are essential for cross-lingual transfer in natural language processing, particularly for low-resource languages. However, evaluating these embeddings typically requires expensive bilingual dictionaries or parallel corpora. This paper introduces a resource-free evaluation metric based on graph modularity, a concept from network science. By measuring the degree to which words cluster by language rather than meaning, this metric serves as a proxy for alignment quality. Our results show a strong negative correlation between modularity and performance on downstream tasks, providing a robust diagnostic tool for unsupervised CLWE development.
Introduction
Cross-lingual word embeddings aim to map words from multiple languages into a single, shared vector space. In an ideal CLWE space, semantically equivalent words (e.g., "dog" in English and "perro" in Spanish) should be positioned closely together. The primary challenge in this field is evaluation: standard intrinsic metrics require gold-standard bilingual dictionaries, which are often unavailable for the very low-resource languages that benefit most from CLWEs.
Furthermore, existing metrics often focus on a small subset of frequent words, failing to capture the structural integrity of the entire embedding space. We address this by identifying a structural "red flag": clustering by language. When words in the same language are consistently closer to each other than to their cross-lingual counterparts, the embedding space is poorly aligned.
Methodology: The Modularity Metric
We propose treating the embedding space as a lexical graph. The construction process involves two main steps:
- Graph Construction: Every word in the vocabulary is treated as a node. Each node is connected to its $k$-nearest neighbors based on cosine similarity.
- Calculating Modularity ($Q$): We apply the standard network science measurement of modularity. In this context, the "communities" or "cliques" are defined by the languages of the words.
The logic is straightforward: high modularity indicates that languages are segregated into distinct clusters within the vector space, signifying poor alignment. Conversely, low modularity suggests that languages are well-mixed, which is a prerequisite for effective cross-lingual transfer. To ensure scalability for large vocabularies, we utilize random projection trees for fast approximate nearest neighbor search.
Experimental Results
We evaluated the modularity metric across six language pairs, including both Indo-European and non-Indo-European families, with a specific focus on low-resource scenarios like Amharic.
Correlation with Task Performance
Our experiments revealed a moderate to strong negative correlation between modularity and performance across several major tasks, including Document Classification ($\rho = -0.665$) and Bilingual Lexical Induction ($\rho = -0.789$). As shown in Figure 2, lower modularity scores consistently correspond to higher precision in lexical induction tasks.
Model Selection and Robustness
Modularity proves particularly useful for unsupervised mapping methods (such as MUSE), which often struggle to converge on distant language pairs like English-Japanese. By using modularity as a validation metric during training, we can identify the "sweet spot" where languages begin to mix correctly. In our tests, using modularity for model selection improved average precision for English-to-Japanese induction from 18.13% to 36.13%.
Discussion and Limitations
While modularity is a powerful diagnostic tool, it is not a complete solution. A critical limitation is the "random trap": a completely random embedding might exhibit low modularity due to perfect mixing, yet it would lack the semantic structure necessary for NLP tasks. Therefore, modularity indicates if languages are mixed, but not necessarily if they are mixed correctly.
Additionally, the metric is sensitive to the choice of $k$ (the number of neighbors). If $k$ is set too high, the graph becomes noisy, and the correlation with task performance diminishes. Future work will explore hybrid metrics that combine modularity with semantic-aware measures to ensure both alignment and accuracy.
Conclusion
Graph modularity provides a robust, resource-free way to evaluate the quality of cross-lingual word embeddings. By focusing on the global neighborhood structure rather than individual word pairs, it offers a unique perspective on alignment that is especially valuable for low-resource language processing.
Summary updated