Bilingual Word Embeddings (BWEs) serve as the backbone for unsupervised machine translation (UMT), enabling models to map lexical items across languages without parallel corpora. However, a persistent hurdle in this field is the assumption of a 1-to-1 mapping. In many language pairs, concepts expressed as a single compound word in one language (e.g., the German Waschmaschine) are realized as multi-word phrases in another (e.g., the English washing machine).
The Challenge of Data Scarcity
Traditional "atomic" approaches attempt to solve this by treating frequent bigrams as single tokens during training. While effective in high-resource scenarios, this strategy fails in low-resource settings. If a target bigram like "washing machine" appears infrequently in a small corpus, the model cannot learn a robust embedding for it, leading to an Out-of-Vocabulary (OOV) problem. This paper, presented at AMTA 2020, proposes a neural decomposition method to bridge this gap.
Methodology: Neural Decomposition
The authors introduce a decomposition-based approach that predicts the constituent parts of a translation rather than searching for a pre-existing bigram vector.
- Architecture: The system uses a two-layer feed-forward neural network. It takes a source unigram embedding ($x_s$) as input and outputs a concatenated vector representing two target unigrams ($[y_{t1}, y_{t2}]$).
- Unsupervised Training: To avoid the need for parallel dictionaries, the model generates training lexicons automatically. This includes "target-to-target" (t2t) tasks where the model learns to decompose a target bigram into its own components, effectively learning the internal logic of the target language's phrase structure.
- Ensembling: The final system combines this decomposition module with a traditional atomic baseline. The atomic model handles frequent, well-represented bigrams, while the decomposition module provides a "generative" fallback for rare or unseen phrases.
Key Results
The researchers evaluated their approach using the new Bilingual Phrase Lexicon Induction (BPLI) test set, simulating low-resource conditions by downsampling training data to sizes of 250K, 500K, and 1M sentences.
As shown in the results above, the ensemble method consistently outperforms the standalone atomic model. The performance gap is most pronounced in the most resource-constrained setting (250K sentences), where the ensemble achieves 15.06% accuracy compared to the baseline's 12.12%. As the corpus size increases, the atomic method begins to catch up as it "sees" more bigrams, but the ensemble maintains its lead.
Handling the "Long Tail"
The primary advantage of the decomposition approach lies in its resilience to the "long tail" of rare phrases. Atomic models rely on memorization; if a bigram is rare, the model has no representation for it.
The data indicates that for very rare bigrams (frequency $\le 10$), the atomic baseline fails entirely (0% accuracy). In contrast, the decomposition module successfully predicts translations by leveraging the compositional meaning of individual words. This capability acts as a vital safety net, allowing the model to "construct" a translation for a phrase it has never encountered as a single unit.
Conclusion and Future Work
By moving from atomic retrieval to neural decomposition, this work provides a robust framework for 1-to-2 word induction in low-resource settings. Integration into full Unsupervised Machine Translation pipelines yielded small but statistically significant improvements in BLEU scores across multiple WMT benchmarks. Future work will aim to expand this to variable-length mappings (1-to-N) and investigate other forms of compositionality, such as Hungarian suffixes or functional words in diverse language families.
Summary updated