TL;DR
This paper tackles the challenge of improving machine translation for rare words and phrases by leveraging monolingual data. They introduce a faster and simpler method to "hallucinate" translation rules for these infrequent phrases by finding similar phrases with known translations using continuous word representations and efficient nearest neighbor search.
What Problem Does This Paper Address?
Statistical machine translation (SMT) systems, while powerful, struggle with translating infrequent words or phrases. This is often due to a lack of sufficient bilingual data to learn reliable translation rules for these "out-of-vocabulary" or rare items. While previous work has explored using abundant monolingual data to induce new translation rules, these methods are often computationally very expensive, making them impractical for large-scale applications. The paper aims to bridge this gap by developing a more efficient and scalable approach to generate translation rules for infrequent phrases using monolingual data.
Methods
The core of the proposed method lies in using continuous representations of words and phrases to find translations for infrequent phrases. Here's a breakdown of their approach:
- Phrase Representations: They use off-the-shelf word embeddings (specifically, the continuous bag-of-words model from Word2Vec) and build phrase vectors by simple element-wise addition. This is a standard technique for creating representations of multi-word units.
- Translation Rule Induction: The key idea is to "hallucinate" translation rules for infrequent (unlabeled) phrases. This is done by identifying labeled phrases (those with known translations in the bilingual data) that are similar in their continuous representation to the unlabeled phrase.
- Global Linear Projection: Initially, they explore learning a single linear projection matrix that maps foreign phrase representations to English phrase representations. This is based on the observation that relative positions of words are preserved across languages.
- Local Linear Projection: Recognizing that a single global projection might not capture all nuances, they propose learning many local linear projections, each specific to a given infrequent foreign phrase. This is achieved by finding the labeled neighbors of the infrequent phrase and learning a projection based on their translations.
- Efficient Nearest Neighbor Search: A crucial component for making this approach scalable is fast retrieval of similar phrases in the continuous space. They compare two approximate k-nearest neighbor (k-NN) search methods:
- Locality Sensitive Hashing (LSH): A popular method for approximate nearest neighbor search.
- Redundant Bit Vectors (RBV): A lesser-known method that they find to be significantly faster and more accurate than LSH for their task, especially in high-dimensional spaces.
Datasets
- Arabic-English Translation:
- Bilingual training data: 685k sentence pairs.
- Tuning and testing sets: NIST MT08 and MT09 data.
- Monolingual data: Primarily from the AFP Gigaword corpus for Arabic and English.
- Urdu-English Translation:
- Bilingual training data: 165k sentence pairs.
- Tuning and testing sets: NIST MT08 and MT09 data.
- Monolingual data: Mined from web crawlers for Urdu, and AFP Gigaword for English.
They collect unigrams and bigrams from tuning/testing sets and monolingual data to form phrase sets. For English, they also use a language model to filter phrases.
Key Insights and Ideas
- Continuous Representations as a Bridge: The paper demonstrates that continuous phrase representations can effectively bridge the gap between monolingual data and the need for translation rules, especially for infrequent phrases.
- "Hallucinating" Rules: The concept of "hallucinating" translation rules by leveraging similarity in continuous space is a novel and intuitive way to tackle the data sparsity problem.
- Local Projections Outperform Global: The finding that local linear projections, tailored to specific infrequent phrases, are significantly more accurate than a single global projection highlights the importance of phrase-specific mappings.
- RBV's Superiority: The significant speed and accuracy advantage of Redundant Bit Vectors (RBV) over Locality Sensitive Hashing (LSH) for approximate k-NN search in this NLP context is a key practical insight. This suggests RBV could be a valuable tool for other high-dimensional NLP tasks.
- Efficiency Gains: The paper emphasizes that their method is orders of magnitude faster than previous semi-supervised approaches, making it practical for real-world deployment.
Key Results
- Arabic-English Translation:
- Achieved a 1.6 BLEU improvement over a standard phrase-based baseline.
- Local Linear Projection (LLP) improved the baseline by 1.3 BLEU on the test set.
- The continuous representation-based SLP improved the baseline by 2.0 BLEU on the tuning set and 1.3 BLEU on the test set.
- The proposed methods are three orders of magnitude faster than existing semi-supervised methods (like PMI-based SLP).
- Urdu-English Translation:
- Improvements were less significant due to smaller datasets, but still showed gains.
- SLP and LLP achieved similar performance to PMI-based SLP but were orders of magnitude faster.
- k-NN Search Performance:
- RBV was 3.6 times faster than LSH and reduced the false negative ratio by 1/3.
Limitations and Future Work
- Limitations:
- The global linear projection (GLP) struggles with infrequent phrases, which is precisely where the method is most needed.
- For some infrequent phrases, there might not be enough labeled neighbors to learn a local linear projection, necessitating a backoff strategy.
- The performance gains are less pronounced on low-resource languages like Urdu due to smaller monolingual corpora.
- Future Work:
- Investigate more elaborate projection schemes, including contextual information and non-linear projections.
- Explore applying redundant bit vectors to other NLP tasks.
- The authors aim to move closer to generating translation rules "on-the-fly" for unseen sentences.
Summary updated