Approaches to Improving Corpus Quality for Statistical Machine Translation

Yu Zhou, Peng Liu, Cheng ZongInt. J. Comput. Process. Orient. Lang 2011 · DOI

TL;DR

This paper tackles the challenge of improving Statistical Machine Translation (SMT) performance by focusing on the quality of bilingual data, not just quantity. They introduce novel methods for filtering noisy sentence pairs from training data and selecting more informative sentences for both training and development sets, demonstrating competitive translation quality with significantly less data.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT) systems heavily rely on large amounts of parallel bilingual text (corpora) to learn translation probabilities. Traditionally, research has focused on acquiring more data, assuming "more is better." However, this approach faces two major hurdles:

  1. Noise: Large corpora often contain misaligned sentence pairs, which introduce errors into the translation model and degrade performance.
  2. Computational Cost & Efficiency: Massive datasets require significant computational resources for training and tuning, making SMT impractical for resource-constrained environments (like mobile devices). Furthermore, large datasets can slow down translation speed.

This paper addresses the need to optimize existing bilingual corpora by improving their quality, thereby achieving better translation performance with smaller, more efficient datasets. They specifically target both the training corpus (for building the translation model) and the development corpus (for tuning translation parameters).

Methods

The authors propose a two-pronged approach to corpus optimization:

1. Noise Filtering for Training Data: * Length Ratio (LR) Policy: This method filters sentence pairs where the ratio of sentence lengths (in words) falls outside a predefined range. The intuition is that parallel sentences should have roughly similar lengths. * Translation Ratio (TR) Policy: This method uses a bilingual dictionary to estimate how many words in the source sentence have their translations present in the target sentence. A low translation ratio suggests a poor alignment or a non-parallel pair. * Combined Filtering: They apply both LR and TR policies sequentially for more robust noise removal.

2. Data Selection for Training and Development Data: * Weighted-Phrase Method (for Training Data): * Phrase Weighting: They assign a weight to each phrase based on its information content (using -log p(f)) and its length (using sqrt(length)). Longer phrases are considered more informative. * Sentence Weighting: Sentences are weighted based on the sum of weights of unseen phrases they contain. This prioritizes sentences that introduce new linguistic information not yet covered by the corpus. * Compact Corpus Construction: Sentences are selected to build a compact training set based on these weights, aiming to retain informative content while reducing size. * Phrase-Coverage-Based Method (PCBM) (for Development Data): * Phrase Weighting: Similar to the training data method, but all phrases within a sentence are considered, not just unseen ones. They limit phrase length to four words to mitigate data sparseness. * Sentence Scoring: Sentences are scored based on the sum of weights of their constituent phrases. * Selection: Sentences with higher scores are prioritized for the development set. They explore using monolingual phrases (Chinese or English) and bilingual phrases. * Structure-Coverage-Based Method (SCBM) (for Development Data): * Phrase-Structure Parsing: The development corpus is parsed into phrase-structure trees. * Subtree Weighting: Subtrees (with depths between 2 and 4) are weighted based on their information content (-log p(t)) and depth (sqrt(depth)). * Sentence Scoring: Sentences are scored based on the sum of weights of the subtrees they contain. * Selection: Sentences with higher scores are selected.

Datasets

  • CWMT 2008 Corpus: Used for experiments on training data processing (Chinese-to-English).
  • CWMT 2009 Corpus: Used for experiments on development data selection (Chinese-to-English and English-to-Chinese).
  • IWSLT 2009 Corpora: Used for experiments on development data selection (Chinese-to-English and English-to-Chinese).

The paper does not introduce new benchmarks but rather applies its methods to existing, standard corpora for machine translation evaluation.

Key Insights and Ideas

  • Quality over Quantity: The paper strongly advocates for prioritizing corpus quality over sheer size, demonstrating that well-curated, smaller corpora can outperform larger, noisier ones.
  • Informative Sentences Matter: The core idea is that not all sentences contribute equally to translation model learning or parameter tuning. Identifying and selecting "informative" sentences is key.
  • Bilingual Features for Development Data: Using bilingual phrase information (PCBM with Ch+En) for development set selection leads to more robust and higher-performing translation parameters compared to monolingual features or structure-based methods.
  • Over-selection is Detrimental: For development sets, using too much data can actually hurt performance by introducing redundancy and noise, leading to suboptimal parameter tuning.
  • TR Policy's Effectiveness: The Translation Ratio (TR) policy for noise filtering proves to be more effective than the Length Ratio (LR) policy, likely due to its reliance on bilingual dictionary information.

Key Results

  • Training Data Noise Filtering: Combining LR and TR policies filtered out 0.6 million words from the CWMT 2008 corpus with minimal BLEU score decline (0.04% loss) or even improvement (0.21% gain with TR alone).
  • Training Data Selection: Using only half the training data with their weighted-phrase method (WP) achieved a BLEU score of 0.2060, which was only 0.72% lower than using all the data (0.2132), while covering 92.3% of words compared to the baseline's 45.9%.
  • Development Data Selection: Both PCBM and SCBM methods outperformed the baseline (random selection) on CWMT 2009 and IWSLT 2009 tasks. PCBM with bilingual phrase coverage (Ch+En) consistently yielded the best results.
  • Optimal Development Set Size: In some cases, using a smaller development set (e.g., 10k words) resulted in better performance than using a larger one (e.g., 12k words), highlighting the importance of selection over sheer size.

Limitations and Future Work

  • SCBM Performance: The Structure-Coverage-Based Method (SCBM) performed less effectively than PCBM. The authors attribute this to parser errors and data sparseness in syntactic structures, and the fact that their SMT engine is phrase-based, not fully leveraging tree structures.
  • Parser Precision: The reliance on parsers for SCBM is a limitation due to their inherent imperfections.
  • Future Work: The authors suggest further exploration into combining phrase and structure features, and potentially developing more robust parsing techniques or alternative methods for capturing structural information. They also implicitly suggest that their methods can be applied to other SMT architectures.

Summary updated

BibTeX
@article{zhou2011approaches,
  author = {Yu Zhou and Peng Liu and Cheng Zong},
  title = {{Approaches to Improving Corpus Quality for Statistical Machine Translation}},
  journal = {Int. J. Comput. Process. Orient. Lang.},
  year = {2011},
  doi = {10.1142/s1793840611002395},
}
Figure