Probabilistic Models of Cross-Lingual Semantic Similarity in Context Based on Latent Cross-Lingual Concepts Induced from Comparable Data

Ivan Vulic, Marie-Francine MoensEMNLP 2014 · OpenReview

Abstract

We propose the first probabilistic approach to modeling cross-lingual semantic similarity (CLSS) in context which requires only comparable data. The approach relies on an idea of projecting words and sets of words into a shared latent semantic space spanned by language-pair independent latent semantic concepts (e.g., crosslingual topics obtained by a multilingual topic model). These latent cross-lingual concepts are induced from a comparable corpus without any additional lexical resources. Word meaning is represented as a probability distribution over the latent concepts, and a change in meaning is represented as a change in the distribution over these latent concepts. We present new models that modulate the isolated out-of-context word representations with contextual knowledge. Results on the task of suggesting word translations in context for 3 language pairs reveal the utility of the proposed contextualized models of crosslingual semantic similarity.

Introduction

Most existing models for Cross-Lingual Semantic Similarity (CLSS) are context-insensitive, providing a global, static similarity score between words in different languages. For instance, a standard model might associate the English word match with both the Spanish partido (game) and cerilla (fire-starter) without the ability to distinguish between them based on usage. While models trained on parallel corpora can address this, such data is scarce. This research explores using more abundant comparable corpora—texts on similar topics but not direct translations—to build context-aware models that can precisely suggest translations based on surrounding text.

Methodology: Latent Cross-Lingual Concepts

The proposed framework projects words and contexts into a shared latent semantic space spanned by Latent Cross-Lingual Concepts, induced via Multilingual Probabilistic Topic Modeling (e.g., bilingual LDA). The core innovation is the "fusion" of a word's global meaning with its local context.

Figure 1: Comparison between context-insensitive and context-sensitive word representations in a latent semantic space.

The authors introduce three fusion models:

  1. Direct-Fusion: Recalculates the topic distribution based on both the target word and context using Bayes' rule.
  2. Smoothed-Fusion: Interpolates context-based representations with global representations, acting as a "back-off" for sparse contexts.
  3. Late-Fusion: Linearly combines separate similarity scores for context and word.

A critical step is Context Sorting and Pruning, where only the top $M$ (e.g., $M=3$) most semantically related words are kept as context, filtering out uninformative "noise."

Experimental Setup and Datasets

The models were trained on comparable Wikipedia article pairs for Spanish-English, Italian-English, and Dutch-English. Evaluation was conducted on a new benchmark of 15 polysemous nouns per language, with 24 sentences per noun (360 sentences total per language).

Figure 3: Examples of the polysemous word 'calcio' in Italian and its context-dependent English translations.

As shown in the table above, words like "calcio" can mean "stock" (of a gun), "calcium," or "football" depending on the sentence. The task is to correctly identify the English translation given the Italian context.

Key Results and Insights

The context-sensitive models significantly outperformed context-insensitive baselines:

  • Accuracy: Context-insensitive models achieved ~40-43% accuracy, while the best context-aware models (typically Smoothed-Fusion) reached 70-79%.
  • Context Pruning: Using only the 3 most related words as context performed better than using entire sentences, proving that "less is more" when avoiding semantic noise.
  • Comparable Data Utility: The results demonstrate that high-quality, context-aware semantic models can be built without perfectly aligned parallel translations, facilitating NLP development for lower-resource languages.

Conclusion and Future Work

This work establishes that probabilistic models can effectively leverage comparable data for context-aware CLSS. Future research could move beyond the "bag-of-words" assumption by incorporating syntactic dependencies and exploring wider contexts (paragraphs) or more distant language pairs like English-Chinese.

Summary updated

Abstract

We propose the first probabilistic approach to modeling cross-lingual semantic similarity (CLSS) in context which requires only comparable data. The approach relies on an idea of projecting words and sets of words into a shared latent semantic space spanned by language-pair independent latent semantic concepts (e.g., crosslingual topics obtained by a multilingual topic model). These latent cross-lingual concepts are induced from a comparable corpus without any additional lexical resources. Word meaning is represented as a probability distribution over the latent concepts, and a change in meaning is represented as a change in the distribution over these latent concepts. We present new models that modulate the isolated out-ofcontext word representations with contextual knowledge. Results on the task of suggesting word translations in context for 3 language pairs reveal the utility of the proposed contextualized models of crosslingual semantic similarity.
BibTeX
@inproceedings{vulic2014probabilistic,
  author = {Ivan Vulic and Marie-Francine Moens},
  title = {{Probabilistic Models of Cross-Lingual Semantic Similarity in Context Based on Latent Cross-Lingual Concepts Induced from Comparable Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  url = {https://openreview.net/forum?id=HJWwSMfdbH},
}
Figure