Word's Vector Representations meet Machine Translation

Eva Martínez Garcia, Jörg Tiedemann, Cristina España-Bonet, Lluís Màrquez

TL;DR

This paper explores how to improve machine translation (MT) by leveraging bilingual word embeddings. They propose a method to create these embeddings directly from parallel corpora, aiming to better handle word ambiguity and improve translation coherence, especially for words with multiple meanings.

What Problem Does This Paper Address?

Modern machine translation systems, while impressive, often operate at the sentence level, neglecting the broader document context. This can lead to inconsistencies and errors, particularly when translating words that have multiple meanings (polysemy). Standard word embedding techniques, like the popular word2vec, also struggle with this lexical ambiguity, conflating different senses of a word into a single representation. The authors aim to bridge this gap by developing bilingual word embeddings that capture semantic relationships across languages, specifically to address word ambiguity in MT.

Methods

The core of the paper is the adaptation of the Continuous Bag-of-Words (CBOW) algorithm for bilingual word representations. Instead of training on monolingual data, they use word-aligned parallel corpora (pairs of sentences that are translations of each other). By applying CBOW to these aligned word pairs, they create a single model that captures semantic information for both the source and target words simultaneously. This differs from other bilingual approaches that build separate models for each language. They then evaluate these bilingual embeddings on two tasks: predicting semantically related words and cross-lingual lexical substitution.

Datasets

  • Training Data: Freely available parallel corpora for English and Spanish, including EuropalV7, United Nations and Multilingual United Nations, and Subtitles2012, all sourced from the Opus site. They used approximately 600 million words for training.
  • Evaluation Data:
    • Semantic Model Evaluation: A bilingual test set of 19,544 questions, adapted from a monolingual set by Mikolov et al. (2013c).
  • Cross-Lingual Lexical Substitution: A test set of news data from the News Commentary corpus of 2011.

Key Insights and Ideas

  • Direct Bilingual Embeddings: The paper argues for building bilingual word embeddings directly from parallel corpora, rather than combining monolingual embeddings. This approach aims to capture cross-lingual semantic nuances more effectively.
  • Handling Lexical Ambiguity: The primary motivation is to tackle word ambiguity, a known limitation of standard word embeddings, by creating representations that can differentiate between word senses in a bilingual context.
  • CBOW for Bilingual Context: Adapting the CBOW architecture to word pairs from parallel corpora is a novel application, suggesting that the "predict-a-word-from-context" strategy can be extended to cross-lingual scenarios.
  • Bottlenecks in Quality: The authors highlight that the quality of word alignments and the underlying parallel corpora significantly impact the performance of their bilingual models. Noise in these sources can lead to lost vocabulary and reduced accuracy.

Key Results

  • Semantic Model Accuracy: For predicting semantically related words, their bilingual model (bi_en-es) achieved an overall accuracy of 23.68% on a test set. This was lower than the monolingual English model (32.47%) but higher than the monolingual Spanish model (10.24%). The authors attribute the lower bilingual accuracy to potential noise in translations and word alignments.
  • Cross-Lingual Lexical Substitution: In this task, their bilingual models achieved an accuracy of 62.4% for retranslating ambiguous words, which was slightly better than using the most frequent translation (59.8%). This modest improvement suggests potential benefits for MT.

Limitations and Future Work

  • Limitations:
    • The accuracy of the bilingual models is affected by the quality of word alignments and the inherent noise in automatically translated corpora.
  • A portion of the vocabulary and translation pairs are lost during the training process due to alignment issues, limiting the models' coverage.
  • Future Work:
    • Investigating different alignment heuristics to improve the quality of word-pair extraction.
  • Developing new features based on these semantic models to integrate them into state-of-the-art SMT systems like Moses or discourse-oriented decoders like Docent.

Summary updated

BibTeX
@inproceedings{garcia2014word,
  author = {Eva Martínez Garcia and Jörg Tiedemann and Cristina España-Bonet and Lluís Màrquez},
  title = {{Word's Vector Representations meet Machine Translation}},
  booktitle = {SSST @ EMNLP},
  year = {2014},
  doi = {10.3115/v1/w14-4015},
}
Figure