Improving Statistical Machine Translation for a Resource-Poor Language Using Related Resource-Rich Languages

Preslav Ivanov Nakov, Hwee Tou NgJAIR 2012 · DOI · OpenReview

TL;DR

This paper introduces a novel, language-independent approach to improve statistical machine translation (SMT) for resource-poor languages by leveraging parallel data from closely related, resource-rich languages. The method achieves significant gains in translation quality (BLEU scores) and drastically reduces the amount of "real" training data needed, making SMT more accessible for a wider range of languages.

What Problem Does This Paper Address?

Building high-quality Statistical Machine Translation (SMT) systems traditionally requires vast amounts of parallel text (bi-texts) for a given language pair. However, for the vast majority of the world's languages, such extensive parallel corpora are scarce, rendering them "resource-poor" from an SMT perspective. This paper tackles the challenge of improving SMT for these resource-poor languages by exploiting the readily available parallel data of closely related, resource-rich languages. The core problem is how to effectively transfer knowledge and improve translation quality when direct parallel data is limited.

Methods

The paper proposes a language-independent approach that leverages a resource-rich language (X₂) to improve translation from a resource-poor language (X₁) to a target language (Y), given a small X₁-Y bi-text and a larger X₂-Y bi-text. The key idea is to exploit the similarities between X₁ and X₂ in terms of vocabulary, spelling, word order, and syntax.

The authors explore several strategies:

  • Bi-text Concatenation: Simply combining the X₁-Y and X₂-Y bi-texts. They investigate variations like repeating the X₁-Y bi-text (catxk) to balance its influence and using the combined bi-text solely for word alignment (catxk:align).
  • Phrase Table Combination: Building separate phrase tables for X₁-Y and X₂-Y and then combining them using methods like:
  • Two-tables: Using them as alternative decoding paths.
  • Interpolation: Linearly interpolating the phrase translation probabilities.
  • Merge: Concatenating phrase pairs, prioritizing those from X₁-Y, and using additional features to indicate the source of each pair.
  • Proposed Hybrid Approach (Merge): This approach combines the strengths of improved word alignments (from catxk:align) and increased lexical coverage (from catx1). It involves building a phrase table from a balanced concatenation (Brep) with truncated alignments, and another from a simple concatenation (Bcat), then merging these two tables with priority given to the first and using extra features.
  • Transliteration: For languages with similar spelling but systematic differences (like Portuguese and Spanish), they develop a character-level SMT system to transliterate words from the resource-rich language to match the resource-poor language's spelling. This transliterated data is then used in the SMT pipeline.
  • Cognate Extraction: They also explore combining their method with techniques that extract likely cognates between the source and target languages and add them as additional training data.
  • Phrase Table Pivoting: They investigate combining their approach with phrase table pivoting techniques.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Indonesian-English (in-en):
    • Train: 28,383 sentence pairs (0.8M, 0.9M words)
    • Dev: 2,000 sentence pairs
    • Test: 2,000 sentence pairs
    • Monolingual English (enin): 5.1M words
  • Malay-English (ml-en):
    • Train: 190,503 sentence pairs (5.4M, 5.8M words)
    • Dev: 2,000 sentence pairs
    • Test: 2,000 sentence pairs
    • Monolingual English (enml): 27.9M words
  • Spanish-English (es-en):
    • Train: 1,240,518 sentence pairs (35.7M, 34.6M words)
    • Dev: 2,000 sentence pairs
    • Test: 2,000 sentence pairs
    • Monolingual English (enes:pt): 45.3M words (shared with pt-en and it-en)
  • Portuguese-English (pt-en):
    • Train: 1,230,038 sentence pairs (35.9M, 34.6M words)
    • Dev: 2,000 sentence pairs
    • Test: 2,000 sentence pairs
    • Monolingual English (enes:pt): 45.3M words
  • Italian-English (it-en):
    • Train: 1,565,885 sentence pairs (43.5M, 44.1M words)
    • Dev: 2,000 sentence pairs
    • Test: 2,000 sentence pairs
    • Monolingual English (enes:it): 45.3M words

The Spanish, Portuguese, and Italian bi-texts are derived from the Europarl corpus. The Indonesian and Malay bi-texts were constructed from comparable web-downloaded texts.

Key Insights and Ideas

  • Leveraging Linguistic Similarity is Key: The core insight is that even small amounts of parallel data from a closely related language can significantly boost SMT performance for a resource-poor language. This is due to shared vocabulary, syntax, and spelling patterns.
  • Transliteration is Crucial for Similar-but-Different Languages: For languages with systematic spelling variations (e.g., Portuguese and Spanish), transliterating the auxiliary language's data to match the target language's spelling dramatically improves translation quality, often doubling the BLEU score.
  • Phrase Table Merging Outperforms Simple Concatenation: Combining phrase tables, especially with a strategy that prioritizes the original language's data and uses additional features, yields better results than simply concatenating bi-texts. This allows for better control over the influence of the auxiliary data.
  • Improved Word Alignments and Lexical Coverage are Synergistic: The proposed hybrid approach, which isolates and then combines the benefits of improved word alignments (from catxk:align) and increased lexical coverage (from catx1), leads to the best overall performance.
  • Less Similar Languages Can Still Be Useful: While closer linguistic relatives provide greater gains, even more dissimilar languages (like Italian for Spanish) can offer improvements, especially when the original resource-poor data is very limited.
  • "Real" Data Reduction is Substantial: The method effectively reduces the need for large amounts of direct parallel data, cutting the required "real" training data by a factor of 2-5.

Key Results

  • Indonesian→English SMT (using Malay): Achieved an absolute gain of up to 1.35 BLEU points over the baseline. The proposed "Our approach" (merging phrase tables) consistently outperformed other concatenation and merging strategies.
  • Spanish→English SMT (using Portuguese): Achieved an absolute gain of up to 3.37 BLEU points. Transliteration was found to be critical, doubling the absolute gain.
  • Data Efficiency: The method significantly reduces the amount of required "real" training data, cutting it by a factor of 2-5. For example, 10K "real" Spanish-English sentences with 160K Portuguese-English sentences (and transliteration) achieved comparable results to 40K "real" sentences for the baseline.
  • Combinations with Other Techniques: Combining the proposed method with cognate extraction and phrase table pivoting also yielded statistically significant improvements, particularly for smaller datasets.

Limitations and Future Work

  • Data Size Imbalance: Simple concatenation can be problematic if the auxiliary bi-text is much larger, dominating word alignment and phrase extraction. The proposed methods aim to mitigate this.
  • Transliteration Complexity: While effective, transliteration requires building a separate SMT system and can be computationally intensive.
  • Language Similarity Matters: While less similar languages can still be useful, the gains are generally larger with closer linguistic relatives.
  • Future Work:
    • Better utilization of multi-lingual parallel corpora (e.g., using Spanish-Portuguese-English corpora more holistically).
  • Exploring auxiliary languages related to the target language.
  • Developing more sophisticated methods to bridge the gap between source and auxiliary languages beyond simple transliteration.

Summary updated

Abstract

We propose a novel language-independent approach for improving machine translation for resource-poor languages by exploiting their similarity to resource-rich ones. More precisely, we improve the translation from a resource-poor source language X_1 into a resource-rich language Y given a bi-text containing a limited number of parallel sentences for X_1-Y and a larger bi-text for X_2-Y for some resource-rich language X_2 that is closely related to X_1. This is achieved by taking advantage of the opportunities that vocabulary overlap and similarities between the languages X_1 and X_2 in spelling, word order, and syntax offer: (1) we improve the word alignments for the resource-poor language, (2) we further augment it with additional translation options, and (3) we take care of potential spelling differences through appropriate transliteration. The evaluation for Indonesian- >English using Malay and for Spanish -> English using Portuguese and pretending Spanish is resource-poor shows an absolute gain of up to 1.35 and 3.37 BLEU points, respectively, which is an improvement over the best rivaling approaches, while using much less additional data. Overall, our method cuts the amount of necessary "real training data by a factor of 2--5.
BibTeX
@article{nakov2012improving,
  author = {Preslav Ivanov Nakov and Hwee Tou Ng},
  title = {{Improving Statistical Machine Translation for a Resource-Poor Language Using Related Resource-Rich Languages}},
  journal = {J. Artif. Intell. Res.},
  year = {2012},
  doi = {10.1613/jair.3540},
}
Figure