TL;DR
This paper introduces a novel, language-independent approach to improve statistical machine translation (SMT) for resource-poor languages by leveraging parallel data from closely related, resource-rich languages. The method achieves significant gains in translation quality (BLEU scores) and drastically reduces the amount of "real" training data needed, making SMT more accessible for a wider range of languages.
What Problem Does This Paper Address?
Building high-quality Statistical Machine Translation (SMT) systems traditionally requires vast amounts of parallel text (bi-texts) for a given language pair. However, for the vast majority of the world's languages, such extensive parallel corpora are scarce, rendering them "resource-poor" from an SMT perspective. This paper tackles the challenge of improving SMT for these resource-poor languages by exploiting the readily available parallel data of closely related, resource-rich languages. The core problem is how to effectively transfer knowledge and improve translation quality when direct parallel data is limited.
Methods
The paper proposes a language-independent approach that leverages a resource-rich language (X₂) to improve translation from a resource-poor language (X₁) to a target language (Y), given a small X₁-Y bi-text and a larger X₂-Y bi-text. The key idea is to exploit the similarities between X₁ and X₂ in terms of vocabulary, spelling, word order, and syntax.
The authors explore several strategies:
- Bi-text Concatenation: Simply combining the X₁-Y and X₂-Y bi-texts. They investigate variations like repeating the X₁-Y bi-text (
catxk) to balance its influence and using the combined bi-text solely for word alignment (catxk:align). - Phrase Table Combination: Building separate phrase tables for X₁-Y and X₂-Y and then combining them using methods like:
- Two-tables: Using them as alternative decoding paths.
- Interpolation: Linearly interpolating the phrase translation probabilities.
- Merge: Concatenating phrase pairs, prioritizing those from X₁-Y, and using additional features to indicate the source of each pair.
- Proposed Hybrid Approach (Merge): This approach combines the strengths of improved word alignments (from
catxk:align) and increased lexical coverage (fromcatx1). It involves building a phrase table from a balanced concatenation (Brep) with truncated alignments, and another from a simple concatenation (Bcat), then merging these two tables with priority given to the first and using extra features. - Transliteration: For languages with similar spelling but systematic differences (like Portuguese and Spanish), they develop a character-level SMT system to transliterate words from the resource-rich language to match the resource-poor language's spelling. This transliterated data is then used in the SMT pipeline.
- Cognate Extraction: They also explore combining their method with techniques that extract likely cognates between the source and target languages and add them as additional training data.
- Phrase Table Pivoting: They investigate combining their approach with phrase table pivoting techniques.
Datasets
The paper utilizes several datasets for training and evaluation:
- Indonesian-English (in-en):
- Train: 28,383 sentence pairs (0.8M, 0.9M words)
- Dev: 2,000 sentence pairs
- Test: 2,000 sentence pairs
- Monolingual English (enin): 5.1M words
- Malay-English (ml-en):
- Train: 190,503 sentence pairs (5.4M, 5.8M words)
- Dev: 2,000 sentence pairs
- Test: 2,000 sentence pairs
- Monolingual English (enml): 27.9M words
- Spanish-English (es-en):
- Train: 1,240,518 sentence pairs (35.7M, 34.6M words)
- Dev: 2,000 sentence pairs
- Test: 2,000 sentence pairs
- Monolingual English (enes:pt): 45.3M words (shared with pt-en and it-en)
- Portuguese-English (pt-en):
- Train: 1,230,038 sentence pairs (35.9M, 34.6M words)
- Dev: 2,000 sentence pairs
- Test: 2,000 sentence pairs
- Monolingual English (enes:pt): 45.3M words
- Italian-English (it-en):
- Train: 1,565,885 sentence pairs (43.5M, 44.1M words)
- Dev: 2,000 sentence pairs
- Test: 2,000 sentence pairs
- Monolingual English (enes:it): 45.3M words
The Spanish, Portuguese, and Italian bi-texts are derived from the Europarl corpus. The Indonesian and Malay bi-texts were constructed from comparable web-downloaded texts.
Key Insights and Ideas
- Leveraging Linguistic Similarity is Key: The core insight is that even small amounts of parallel data from a closely related language can significantly boost SMT performance for a resource-poor language. This is due to shared vocabulary, syntax, and spelling patterns.
- Transliteration is Crucial for Similar-but-Different Languages: For languages with systematic spelling variations (e.g., Portuguese and Spanish), transliterating the auxiliary language's data to match the target language's spelling dramatically improves translation quality, often doubling the BLEU score.
- Phrase Table Merging Outperforms Simple Concatenation: Combining phrase tables, especially with a strategy that prioritizes the original language's data and uses additional features, yields better results than simply concatenating bi-texts. This allows for better control over the influence of the auxiliary data.
- Improved Word Alignments and Lexical Coverage are Synergistic: The proposed hybrid approach, which isolates and then combines the benefits of improved word alignments (from
catxk:align) and increased lexical coverage (fromcatx1), leads to the best overall performance. - Less Similar Languages Can Still Be Useful: While closer linguistic relatives provide greater gains, even more dissimilar languages (like Italian for Spanish) can offer improvements, especially when the original resource-poor data is very limited.
- "Real" Data Reduction is Substantial: The method effectively reduces the need for large amounts of direct parallel data, cutting the required "real" training data by a factor of 2-5.
Key Results
- Indonesian→English SMT (using Malay): Achieved an absolute gain of up to 1.35 BLEU points over the baseline. The proposed "Our approach" (merging phrase tables) consistently outperformed other concatenation and merging strategies.
- Spanish→English SMT (using Portuguese): Achieved an absolute gain of up to 3.37 BLEU points. Transliteration was found to be critical, doubling the absolute gain.
- Data Efficiency: The method significantly reduces the amount of required "real" training data, cutting it by a factor of 2-5. For example, 10K "real" Spanish-English sentences with 160K Portuguese-English sentences (and transliteration) achieved comparable results to 40K "real" sentences for the baseline.
- Combinations with Other Techniques: Combining the proposed method with cognate extraction and phrase table pivoting also yielded statistically significant improvements, particularly for smaller datasets.
Limitations and Future Work
- Data Size Imbalance: Simple concatenation can be problematic if the auxiliary bi-text is much larger, dominating word alignment and phrase extraction. The proposed methods aim to mitigate this.
- Transliteration Complexity: While effective, transliteration requires building a separate SMT system and can be computationally intensive.
- Language Similarity Matters: While less similar languages can still be useful, the gains are generally larger with closer linguistic relatives.
- Future Work:
- Better utilization of multi-lingual parallel corpora (e.g., using Spanish-Portuguese-English corpora more holistically).
- Exploring auxiliary languages related to the target language.
- Developing more sophisticated methods to bridge the gap between source and auxiliary languages beyond simple transliteration.
Summary updated