Are All Good Word Vector Spaces Isomorphic?

Ivan Vulic, Sebastian Ruder, Anders Søgaard

Abstract

Existing algorithms for aligning cross-lingual word vector spaces assume that vector spaces are approximately isomorphic. As a result, they perform poorly or fail completely on non-isomorphic spaces. Such non-isomorphism has been hypothesised to result from typological differences between languages. In this work, we ask whether non-isomorphism is also crucially a sign of degenerate word vector spaces. We present a series of experiments across diverse languages which show that variance in performance across language pairs is not only due to typological differences, but can mostly be attributed to the size of the monolingual resources available, and to the properties and duration of monolingual training (e.g. "under-training").

Introduction

The creation of cross-lingual word embeddings typically involves training monolingual vector spaces independently and then learning a linear transformation to map one onto the other. This approach rests on the fundamental assumption of isomorphism: the idea that the geometric structures of word vector spaces are similar across languages. While this assumption holds for high-resource, similar languages like English and Spanish, it frequently breaks down for distant or low-resource languages.

Historically, the NLP community has attributed these failures to typological differences—the notion that different grammars and morphologies naturally lead to different conceptual organizations. However, this paper investigates an alternative hypothesis: is the lack of isomorphism a result of linguistic distance, or is it an artifact of "degenerate" vector spaces caused by insufficient training data and compute?

Figure 1: BLI scores (Accuracy) vs. the number of tokens in Wikipedia across various languages.

Methodology

To isolate the effects of data scarcity from linguistic typology, the authors simulated low-resource conditions using high-resource languages (English, Spanish, Arabic, and Japanese). They systematically controlled for several factors:

  • Corpus Size: Large Wikipedia dumps were subsampled to match the sizes of low-resource languages (ranging from 1.3M to 268M tokens).
  • Training Duration: Snapshots of embeddings were taken at various stages of training to model the effects of "under-training."
  • Topical Skew & Morphology: To account for domain biases in low-resource Wikipedias, the authors skewed Spanish data to match the topical distributions of languages like Basque and Quechua. They also used lemmatization to control for morphological complexity.
  • Metrics: Structural similarity was quantified using Relational Similarity (RSIM), alongside standard Bilingual Lexicon Induction (BLI) accuracy.

Key Insights and Results

The experiments revealed that data scarcity and training dynamics are primary drivers of non-isomorphism.

The Impact of Corpus Size

A critical finding is that even aligning English to English fails if one space is trained on a small corpus. This suggests that data scarcity inherently disrupts isomorphism regardless of the language pair. As shown in Figure 4, both BLI performance and structural isomorphism (RSIM) scale logarithmically with corpus size, only beginning to plateau at significant data volumes (10M-50M sentences).

Figure 4: BLI scores and isomorphism (RSIM) as a function of Wikipedia corpus size for English to Spanish, Arabic, and Japanese.

Under-training and Learning Dynamics

Monolingual embeddings trained on small corpora are often "under-trained." The study found that longer training durations strictly improve isomorphism. Interestingly, structural similarity continues to improve even after standard validation metrics and training losses have plateaued. This suggests that the stochastic nature of training (e.g., SGD noise) gradually pushes the vector space toward a more isomorphic state over time.

Confounding Factors: Topical Skew

Low-resource datasets are often skewed by specific contributors or bots. When high-resource data (Spanish) was artificially skewed to match the topics of low-resource languages, alignment performance dropped significantly. This indicates that domain mismatch is a major, often overlooked, factor in alignment failure. By controlling for topical skew and morphology, the performance gap between English-Spanish and English-Basque alignment was reduced by over 50%.

Conclusion

This work demonstrates that the failure of cross-lingual alignment is not solely a consequence of inherent linguistic differences. Instead, it is largely driven by the quality and quantity of monolingual resources. The findings suggest a shift in research priorities: rather than focusing exclusively on complex mapping algorithms, the community should invest in improving monolingual training regimes and data quality for low-resource settings.

Summary updated

Abstract

Existing algorithms for aligning cross-lingual word vector spaces assume that vector spaces are approximately isomorphic. As a result, they perform poorly or fail completely on non-isomorphic spaces. Such non-isomorphism has been hypothesised to result from typological differences between languages. In this work, we ask whether non-isomorphism is also crucially a sign of degenerate word vector spaces. We present a series of experiments across diverse languages which show that variance in performance across language pairs is not only due to typological differences, but can mostly be attributed to the size of the monolingual resources available, and to the properties and duration of monolingual training (e.g. "under-training").
BibTeX
@inproceedings{vulic2020are,
  author = {Ivan Vulic and Sebastian Ruder and Anders Søgaard},
  title = {{Are All Good Word Vector Spaces Isomorphic?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=_cWZe0rUcly},
}
Figure