Abstract: The success of cross-lingual word embeddings (CLWEs) hinges on the assumption of isomorphism—that vector spaces of different languages share a similar geometric structure. When alignment fails, researchers often cite "typological divergence" as the cause. In this work, Vulić et al. demonstrate that this failure is frequently a byproduct of "degenerate" spaces caused by insufficient training data and time. By controlling for these factors, the authors show that even typologically distant languages are significantly more isomorphic than previously believed.
The Isomorphism Assumption
Creating CLWEs typically involves learning a linear transformation to align two monolingual vector spaces. This process assumes that the "shape" of the word cloud in one language (e.g., English) is roughly equivalent to another (e.g., Basque). When these models underperform, the standard explanation has been that languages are simply too different structurally. This paper challenges that narrative, investigating whether the problem lies in the language itself or in the quality of the monolingual models used for low-resource languages.
Experimental Methodology
The authors isolated the effects of data volume and training duration from linguistic typology using several key approaches:
- Isomorphism Metrics: They employed Eigenvector Similarity (EVS), Gromov-Hausdorff (GH) distance, and Relational Similarity (RSIM) to quantify structural matches.
- Controlled Simulations: To test the impact of data, they simulated low-resource scenarios for high-resource languages (e.g., training Spanish on tiny Wikipedia subsets).
- Systematic Variation: They varied corpus size, training updates, topical skew, and morphology (via lemmatization) to pinpoint the drivers of alignment failure.
Key Insights and Ideas
The "Under-training" Trap
A central finding is that most low-resource embeddings are "degenerate"—they lack the stable, isomorphic structure required for alignment because they haven't seen enough data. Crucially, the authors found that even aligning English to English fails if one model is trained on a small dataset, proving that data scarcity, not just linguistic difference, causes structural mismatch.
Isomorphism Beyond Convergence
A critical observation is that vector spaces continue to become more isomorphic even after alignment accuracy (BLI) appears to plateau.
As shown in the figure above, for languages like Arabic and Japanese, the isomorphism metric (RSIM) continues to improve significantly with more training updates, even as the BLI scores (MRR) show diminishing returns. This suggests that for cross-lingual tasks, monolingual models should be trained much longer than is standard for monolingual applications.
Key Results
The researchers found a powerful correlation between the size of a language's training corpus and its alignment potential.
As illustrated above, there is a strong correlation ($\rho = 0.72$) between the number of tokens in a language's Wikipedia and the resulting BLI accuracy. Furthermore, by controlling for corpus size and topical skew, the authors were able to reduce the alignment performance gap between Spanish and Basque from 0.291 to just 0.129. This indicates that while typology matters, its role has been overstated relative to data effects.
Conclusion and Future Work
The study concludes that the difficulty of aligning languages is "almost" entirely a matter of data. While inherent linguistic differences like morphology do play a role, they are often secondary to the "degeneracy" caused by under-training. These findings suggest that before attributing poor performance to a language's unique properties, researchers should ensure models have been given sufficient data and training time to mature. Future work remains to see if these trends hold for more complex contextualized embeddings like BERT or XLM-R.
Summary updated