Phrase-based Unsupervised Machine Translation with Compositional Phrase Embeddings

Maksym Del, Andre Tättar, Mark FishelEMNLP 2018

TL;DR

This paper explores a novel approach to unsupervised machine translation (UMT) by representing phrases not as atomic units, but as compositions of their constituent word embeddings. This method significantly reduces memory requirements and data sparsity issues inherent in traditional phrase-based UMT, while achieving competitive translation performance.

What Problem Does This Paper Address?

Traditional machine translation, especially neural machine translation (NMT), heavily relies on massive parallel corpora (texts with direct translations). However, for many low-resource language pairs, such data is scarce or non-existent. Unsupervised Machine Translation (UMT) aims to bridge this gap by learning to translate using only monolingual data.

Existing UMT methods often struggle with the "phrase" level. Phrase-based UMT (PBUMT) uses phrase tables, but representing these phrases as individual, atomic vocabulary entries (e.g., "New York" as a single token) leads to massive vocabularies that are memory-intensive and suffer from data sparsity (many phrases appear rarely). This paper tackles the challenge of effectively representing phrases in UMT without these limitations.

Methods

The core innovation lies in representing phrases compositionally. Instead of treating a phrase like "New York" as a single, learned embedding, the authors propose learning a model that predicts the phrase's embedding by combining the embeddings of its constituent words ("New" and "York").

The process involves several steps:

  1. Monolingual Embeddings: Standard word embeddings (using FastText) are learned for the source and target languages independently.
  2. Phrase Extraction: Candidate phrases are identified from monolingual corpora using statistical criteria (e.g., likelihood ratio).
  3. Compositional Model Training: A dataset is created where the input is the sequence of word embeddings for a phrase, and the target is the phrase's embedding (learned as an atomic unit in a preliminary step). Various models are trained to predict the phrase embedding from word embeddings, including:
  4. Simple Addition: Summing word vectors.
  5. Attention-based Addition: Weighted sums of word vectors.
  6. Neural Networks: Linear layers, ReLU layers, and multi-layer networks.
  7. LSTM: Leveraging its sequential processing capabilities.
  8. UMT Integration: The learned compositional phrase embeddings are then used within a phrase-based unsupervised machine translation framework. This involves aligning word/phrase embedding spaces and using a standard SMT pipeline for translation.

Datasets

  • English Wikipedia: Used for training compositional phrase embedding models. Specifically, the first 1 billion bytes of English Wikipedia were utilized, containing over 124 million tokens.
  • WMT'18 Unsupervised Translation Task Datasets: Used for training and evaluating the UMT systems for the English-Estonian language pair. Approximately 15 million monolingual sentences were used for each language.

Key Insights and Ideas

  • Compositionality is Key: The paper strongly advocates for representing phrases as compositions of word embeddings rather than atomic units. This aligns with the intuition that many phrases have predictable meanings based on their parts.
  • Simple Summation Works Surprisingly Well: The simplest compositional model, just summing the word vectors, achieved surprisingly good results in predicting phrase embeddings. This suggests that even basic combination methods can capture significant semantic information.
  • Reduced Memory Footprint: By composing embeddings on the fly, the need to store embeddings for every possible phrase is eliminated, drastically reducing memory requirements.
  • Mitigating Data Sparsity: Compositional methods can generate embeddings for phrases that appear rarely or not at all in the training data, overcoming a major limitation of atomic phrase embeddings.
  • LSTM Shows Promise: Among the neural network approaches, the LSTM model demonstrated strong performance, likely due to its ability to capture sequential dependencies and selectively update information.

Key Results

  • Compositional Embeddings Performance: In standalone experiments evaluating the quality of compositional phrase embeddings, models like "Linear" and "MultiNonLinear" achieved high accuracy scores (e.g., ~0.92 for top10 accuracy). Notably, "AddSimple" also performed competitively (~0.94 for top10).
  • UMT Performance: When integrated into the UMT system, the "Bigram-atomic" baseline achieved a significant BLEU score increase of +3.5 for Estonian-English translation compared to the "Unigram" system. The "Bigram-sum" system, using compositional embeddings, also showed improvement over the unigram baseline, though it did not consistently outperform the "Bigram-atomic" system. For English-Estonian, the compositional approach showed promise but didn't consistently surpass atomic baselines.

Limitations and Future Work

  • Limited Phrase Length: Experiments were primarily conducted on phrases of up to two words due to computational constraints.
  • Order Sensitivity: The "AddSimple" model's low top1 accuracy suggests it doesn't inherently capture word order, though this is mitigated at higher top-N values.
  • Further Research Needed: The authors acknowledge that the impact of compositional phrase embeddings on PBUMT requires more extensive study. They suggest exploring more language pairs and conducting targeted experiments with stronger baselines. The potential of compositional embeddings for UMT is still an open research direction.

Summary updated

BibTeX
@inproceedings{del2018phrase,
  author = {Maksym Del and Andre Tättar and Mark Fishel},
  title = {{Phrase-based Unsupervised Machine Translation with Compositional Phrase Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure