A Stronger Baseline for Multilingual Word Embeddings

Philipp Dufter, Hinrich Schütze

Abstract

Word embeddings induced from local context are prevalent in NLP. A simple and effective context-based multilingual embedding learner is Levy et al. (2017)'s S-ID (sentence ID) method. Another line of work induces high-performing multilingual embeddings from concepts (Dufter et al., 2018). In this paper, we propose Co+Co, a simple and scalable method that combines context-based and concept-based learning. From a sentence aligned corpus, concepts are extracted via sampling; words are then associated with their concept ID and sentence ID in embedding learning. This is the first work that successfully combines context-based and concept-based embedding learning. We show that Co+Co performs well for two different application scenarios: the Parallel Bible Corpus (1000+ languages, low-resource) and EuroParl (12 languages, high-resource). Among methods applicable to both corpora, Co+Co performs best in our evaluation setup of six tasks.

Introduction

Multilingual word embeddings (MWEs) map words from multiple languages into a single, shared vector space, enabling cross-lingual transfer for various NLP applications. Traditionally, MWEs are generated using one of two signals: local context (aligned sentences) or global concepts (semantic groupings). While context-based methods like S-ID are robust, they often lack the precision of concept-based approaches. Conversely, concept-based methods can be difficult to scale.

Furthermore, modern deep-learning-based embedding alignment methods, such as MUSE, are highly data-dependent. While they perform well for high-resource language pairs, they often fail in low-resource settings where parallel data is scarce. This paper introduces Co+Co (Context + Concept), a method designed to merge these two signals into a single, scalable pipeline that remains effective across both massively multilingual (1,000+ languages) and high-resource datasets.

Methodology

Co+Co frames multilingual embedding learning as a standard monolingual word2vec problem by generating an artificial corpus. The pipeline integrates two distinct signals:

  1. Context (S-ID): Every word in a parallel corpus is paired with a unique Sentence ID (S-ID). This provides a cross-lingual signal; for instance, if "dog" (English) and "chien" (French) appear in the same aligned sentence, they share the same S-ID.
  2. Concepts (C-ID): Using the Anymalign algorithm, the authors identify "perfect alignments"—words or n-grams that occur exclusively in the same sentences across multiple languages. These form a "concept," and words are paired with their respective Concept IDs (C-ID).

The final artificial corpus is created by concatenating these S-ID and C-ID pairs. This pseudo-corpus is then fed into a standard word2vec skip-gram model. By training on co-occurrences of words with shared IDs, the model naturally pulls semantically related words from different languages into the same region of the vector space.

Illustration of the artificial corpus generation for Co+Co, showing word-to-ID mappings derived from sentences.

Experiments and Data

The method was evaluated on two contrasting datasets:

  • Parallel Bible Corpus (PBC): A massively multilingual, low-resource dataset containing the New Testament in over 1,000 languages.
  • EuroParl: A high-resource dataset covering 12 European languages, derived from parliamentary proceedings and Wikipedia.

Results and Insights

Co+Co proved to be the most stable and high-performing method across the evaluation suite. On the low-resource PBC, Co+Co achieved a roundtrip translation accuracy of 51%, outperforming the context-only S-ID (48%) and concept-only C-ID (43%) baselines.

Evaluation results on the Parallel Bible Corpus across Roundtrip and Sentiment tasks.

Key insights from the experiments include:

  • Complementary Signals: Combining local context (noisy but broad) with global concepts (strict and aggregated) yields superior embeddings.
  • Coverage: Co+Co consistently provided higher vocabulary coverage than competing methods, reducing the number of "out of vocabulary" words in the final embedding space.
  • Neural Mapping Failures: Unsupervised neural mapping methods like MUSE performed poorly on the Bible corpus, often failing to beat naive baselines. This suggests that complex neural models require larger vocabularies than those available in low-resource settings.
  • Dimensionality: For low-resource datasets, lower-dimensional vectors (e.g., 100) often performed better than the standard 300 dimensions, likely due to the limited data available for parameter estimation.

Conclusion

Co+Co offers a simple, scalable, and effective baseline for multilingual word embeddings. By successfully combining context-based and concept-based learning, it provides a robust solution for both high-resource and massively multilingual low-resource scenarios. While specialized methods may occasionally outperform it in specific tasks, Co+Co's versatility makes it a highly practical choice for broad cross-lingual applications.

Summary updated

Abstract

Word embeddings induced from local context are prevalent in NLP. A simple and effective context-based multilingual embedding learner is Levy et al. (2017)'s S-ID (sentence ID) method. Another line of work induces high-performing multilingual embeddings from concepts (Dufter et al., 2018). In this paper, we propose Co+Co, a simple and scalable method that combines context-based and concept-based learning. From a sentence aligned corpus, concepts are extracted via sampling; words are then associated with their concept ID and sentence ID in embedding learning. This is the first work that successfully combines context-based and concept-based embedding learning. We show that Co+Co performs well for two different application scenarios: the Parallel Bible Corpus (1000+ languages, low-resource) and EuroParl (12 languages, high-resource). Among methods applicable to both corpora, Co+Co performs best in our evaluation setup of six tasks.
BibTeX
@misc{dufter2018stronger,
  author = {Philipp Dufter and Hinrich Schütze},
  title = {{A Stronger Baseline for Multilingual Word Embeddings}},
  year = {2018},
  eprint = {1811.00586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.00586},
}
Figure