A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity

Yoshinari Fujinuma, Jordan L. Boyd-Graber, Michael J. PaulACL 2019 · DOI

Abstract

Cross-lingual word embeddings (CLWEs) are essential for cross-lingual transfer in natural language processing, particularly for low-resource languages. However, evaluating these embeddings typically requires expensive bilingual dictionaries or parallel corpora. This paper introduces a resource-free evaluation metric based on graph modularity, a concept from network science. By measuring the degree to which words cluster by language rather than meaning, this metric serves as a proxy for alignment quality. Our results show a strong negative correlation between modularity and performance on downstream tasks, providing a robust diagnostic tool for unsupervised CLWE development.

Introduction

Cross-lingual word embeddings aim to map words from multiple languages into a single, shared vector space. In an ideal CLWE space, semantically equivalent words (e.g., "dog" in English and "perro" in Spanish) should be positioned closely together. The primary challenge in this field is evaluation: standard intrinsic metrics require gold-standard bilingual dictionaries, which are often unavailable for the very low-resource languages that benefit most from CLWEs.

Furthermore, existing metrics often focus on a small subset of frequent words, failing to capture the structural integrity of the entire embedding space. We address this by identifying a structural "red flag": clustering by language. When words in the same language are consistently closer to each other than to their cross-lingual counterparts, the embedding space is poorly aligned.

Methodology: The Modularity Metric

We propose treating the embedding space as a lexical graph. The construction process involves two main steps:

  1. Graph Construction: Every word in the vocabulary is treated as a node. Each node is connected to its $k$-nearest neighbors based on cosine similarity.
  2. Calculating Modularity ($Q$): We apply the standard network science measurement of modularity. In this context, the "communities" or "cliques" are defined by the languages of the words.
Figure 1: Conceptual illustration of low vs. high modularity in cross-lingual word embeddings.

The logic is straightforward: high modularity indicates that languages are segregated into distinct clusters within the vector space, signifying poor alignment. Conversely, low modularity suggests that languages are well-mixed, which is a prerequisite for effective cross-lingual transfer. To ensure scalability for large vocabularies, we utilize random projection trees for fast approximate nearest neighbor search.

Experimental Results

We evaluated the modularity metric across six language pairs, including both Indo-European and non-Indo-European families, with a specific focus on low-resource scenarios like Amharic.

Correlation with Task Performance

Our experiments revealed a moderate to strong negative correlation between modularity and performance across several major tasks, including Document Classification ($\rho = -0.665$) and Bilingual Lexical Induction ($\rho = -0.789$). As shown in Figure 2, lower modularity scores consistently correspond to higher precision in lexical induction tasks.

Figure 2: Negative correlation between modularity and Precision@1 across various language pairs and dimensions.

Model Selection and Robustness

Modularity proves particularly useful for unsupervised mapping methods (such as MUSE), which often struggle to converge on distant language pairs like English-Japanese. By using modularity as a validation metric during training, we can identify the "sweet spot" where languages begin to mix correctly. In our tests, using modularity for model selection improved average precision for English-to-Japanese induction from 18.13% to 36.13%.

Discussion and Limitations

While modularity is a powerful diagnostic tool, it is not a complete solution. A critical limitation is the "random trap": a completely random embedding might exhibit low modularity due to perfect mixing, yet it would lack the semantic structure necessary for NLP tasks. Therefore, modularity indicates if languages are mixed, but not necessarily if they are mixed correctly.

Additionally, the metric is sensitive to the choice of $k$ (the number of neighbors). If $k$ is set too high, the graph becomes noisy, and the correlation with task performance diminishes. Future work will explore hybrid metrics that combine modularity with semantic-aware measures to ensure both alignment and accuracy.

Conclusion

Graph modularity provides a robust, resource-free way to evaluate the quality of cross-lingual word embeddings. By focusing on the global neighborhood structure rather than individual word pairs, it offers a unique perspective on alignment that is especially valuable for low-resource language processing.

Summary updated

BibTeX
@inproceedings{fujinuma2019resource,
  author = {Yoshinari Fujinuma and Jordan L. Boyd-Graber and Michael J. Paul},
  title = {{A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1489},
}
Figure