Abstract
We propose the first probabilistic approach to modeling cross-lingual semantic similarity (CLSS) in context which requires only comparable data. The approach relies on an idea of projecting words and sets of words into a shared latent semantic space spanned by language-pair independent latent semantic concepts (e.g., crosslingual topics obtained by a multilingual topic model). These latent cross-lingual concepts are induced from a comparable corpus without any additional lexical resources. Word meaning is represented as a probability distribution over the latent concepts, and a change in meaning is represented as a change in the distribution over these latent concepts. We present new models that modulate the isolated out-of-context word representations with contextual knowledge. Results on the task of suggesting word translations in context for 3 language pairs reveal the utility of the proposed contextualized models of crosslingual semantic similarity.
Introduction
Most existing models for Cross-Lingual Semantic Similarity (CLSS) are context-insensitive, providing a global, static similarity score between words in different languages. For instance, a standard model might associate the English word match with both the Spanish partido (game) and cerilla (fire-starter) without the ability to distinguish between them based on usage. While models trained on parallel corpora can address this, such data is scarce. This research explores using more abundant comparable corpora—texts on similar topics but not direct translations—to build context-aware models that can precisely suggest translations based on surrounding text.
Methodology: Latent Cross-Lingual Concepts
The proposed framework projects words and contexts into a shared latent semantic space spanned by Latent Cross-Lingual Concepts, induced via Multilingual Probabilistic Topic Modeling (e.g., bilingual LDA). The core innovation is the "fusion" of a word's global meaning with its local context.
The authors introduce three fusion models:
- Direct-Fusion: Recalculates the topic distribution based on both the target word and context using Bayes' rule.
- Smoothed-Fusion: Interpolates context-based representations with global representations, acting as a "back-off" for sparse contexts.
- Late-Fusion: Linearly combines separate similarity scores for context and word.
A critical step is Context Sorting and Pruning, where only the top $M$ (e.g., $M=3$) most semantically related words are kept as context, filtering out uninformative "noise."
Experimental Setup and Datasets
The models were trained on comparable Wikipedia article pairs for Spanish-English, Italian-English, and Dutch-English. Evaluation was conducted on a new benchmark of 15 polysemous nouns per language, with 24 sentences per noun (360 sentences total per language).
As shown in the table above, words like "calcio" can mean "stock" (of a gun), "calcium," or "football" depending on the sentence. The task is to correctly identify the English translation given the Italian context.
Key Results and Insights
The context-sensitive models significantly outperformed context-insensitive baselines:
- Accuracy: Context-insensitive models achieved ~40-43% accuracy, while the best context-aware models (typically Smoothed-Fusion) reached 70-79%.
- Context Pruning: Using only the 3 most related words as context performed better than using entire sentences, proving that "less is more" when avoiding semantic noise.
- Comparable Data Utility: The results demonstrate that high-quality, context-aware semantic models can be built without perfectly aligned parallel translations, facilitating NLP development for lower-resource languages.
Conclusion and Future Work
This work establishes that probabilistic models can effectively leverage comparable data for context-aware CLSS. Future research could move beyond the "bag-of-words" assumption by incorporating syntactic dependencies and exploring wider contexts (paragraphs) or more distant language pairs like English-Chinese.
Summary updated