Multilingual Embeddings Jointly Induced from Contexts and Concepts: Simple, Strong and Scalable

Philipp Dufter, Mengjie Zhao, Hinrich Schütze

Abstract

Word embeddings induced from local context are prevalent in NLP. A simple and effective context-based multilingual embedding learner is Levy et al. (2017)'s S-ID (sentence ID) method. Another line of work induces high-performing multilingual embeddings from concepts (Dufter et al., 2018). In this paper, we propose Co+Co, a simple and scalable method that combines context-based and concept-based learning. From a sentence aligned corpus, concepts are extracted via sampling; words are then associated with their concept ID and sentence ID in embedding learning. This is the first work that successfully combines context-based and concept-based embedding learning. We show that Co+Co performs well for two different application scenarios: the Parallel Bible Corpus (1000+ languages, low-resource) and EuroParl (12 languages, high-resource). Among methods applicable to both corpora, Co+Co performs best in our evaluation setup of six tasks.

Introduction

Multilingual word embeddings—mapping words from different languages into a shared vector space—are essential for cross-lingual tasks like machine translation and sentiment analysis. However, researchers often face a trade-off between context-based methods (like S-ID), which use sentence alignments but may miss broader semantic connections, and concept-based methods (like C-ID), which identify global semantic anchors but ignore local usage context.

Furthermore, many state-of-the-art neural mapping methods struggle to scale to "massively multilingual" scenarios involving thousands of languages with very little data. The authors propose Co+Co (Context + Concept) as a "Goldilocks" method: one that is simple to implement, scales to thousands of languages, and performs strongly regardless of data volume.

Methodology: The Co+Co Approach

The Co+Co method consists of three main steps to bridge the gap between local context and global concepts:

  1. Concept Induction: Using the Anymalign algorithm, the authors extract "concepts" from a sentence-aligned parallel corpus. A concept is a set of words or n-grams that appear exclusively in the same sentences across different languages (perfect alignments).
  2. Corpus Creation: They generate an artificial corpus by combining two signals:
  3. S-ID (Sentence ID): Each word is paired with its sentence identifier, anchoring it to a local context.
  4. C-ID (Concept ID): Each word is paired with its induced concept identifier, acting as a global semantic anchor.
  5. Embedding Learning: A standard word2vec (skip-gram) model is trained on the combined artificial corpus. By treating sentence IDs and concept IDs as "words" in the same space as the actual vocabulary, the model naturally learns to cluster translations together.
Illustration of the S-ID and C-ID data formats derived from aligned sentences.

Experimental Setup

The authors evaluated Co+Co on two distinct benchmarks:

  • Parallel Bible Corpus (PBC): A massively multilingual, low-resource dataset covering over 1,000 languages.
  • EuroParl: A high-resource dataset covering 12 languages from European Parliament proceedings.

Key Results

Co+Co proved to be the most stable performer across all tasks. While specialized methods might beat it on specific datasets, Co+Co was the only method to rank in the top tier for both the 1,000-language Bible corpus and the 12-language EuroParl corpus.

Performance comparison of Co+Co against various baselines across Roundtrip and Sentiment tasks.

As shown in the results table above, Co+Co (row 5) demonstrates competitive performance across Roundtrip translation and Sentiment analysis metrics. In a final performance overview across six tasks, Co+Co achieved the best average rank (1.67) compared to S-ID (2.33) and N(t) (2.00).

Discussion and Insights

  • Complementary Signals: The core innovation is that context and concepts provide non-overlapping information. S-ID captures local behavior, while C-ID captures global semantic equivalence. Combining them yields a more robust signal.
  • Simplicity Scales: Unlike complex neural architectures requiring GPUs, Co+Co relies on word2vec, making it computationally efficient enough to handle 1,000+ languages simultaneously.
  • Dimensionality Matters: For low-resource data like the Bible, lower dimensionality (e.g., 100) is actually better. High-dimensional spaces are harder to populate accurately when data is sparse, leading to noise that hurts translation performance.

Conclusion

Co+Co is a simple, scalable, and effective method for inducing multilingual embeddings. By successfully combining context-based and concept-based signals, it provides a versatile solution for both high-resource and massively multilingual low-resource settings. Future work may explore integrating these joint signals into the pre-training of large-scale multilingual transformer models.

Summary updated

Abstract

Word embeddings induced from local context are prevalent in NLP. A simple and effective context-based multilingual embedding learner is Levy et al. (2017)'s S-ID (sentence ID) method. Another line of work induces high-performing multilingual embeddings from concepts (Dufter et al., 2018). In this paper, we propose Co+Co, a simple and scalable method that combines context-based and concept-based learning. From a sentence aligned corpus, concepts are extracted via sampling; words are then associated with their concept ID and sentence ID in embedding learning. This is the first work that successfully combines context-based and concept-based embedding learning. We show that Co+Co performs well for two different application scenarios: the Parallel Bible Corpus (1000+ languages, low-resource) and EuroParl (12 languages, high-resource). Among methods applicable to both corpora, Co+Co performs best in our evaluation setup of six tasks.
BibTeX
@misc{dufter2018multilingual,
  author = {Philipp Dufter and Mengjie Zhao and Hinrich Schütze},
  title = {{Multilingual Embeddings Jointly Induced from Contexts and Concepts: Simple, Strong and Scalable}},
  year = {2018},
  eprint = {1811.00586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.00586},
}
Figure