Example-based Rescoring of Statistical Machine Translation Output

Michael Paul, Eiichiro Sumita, Seiichi YamamotoNAACL 2004

TL;DR

This paper introduces an "example-based rescoring" method to improve statistical machine translation (SMT) by using retrieved translation examples to validate SMT output. By penalizing translations that deviate significantly from similar examples, this approach significantly boosts translation accuracy, outperforming standard SMT methods.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT) systems aim to find the most probable translation of a source sentence by combining translation models (TM) and language models (LM). However, SMT can struggle due to two main issues:

  1. Data Sparseness: When estimating model parameters, insufficient parallel data can lead to inaccurate probabilities.
  2. Search Errors: The decoding process, which generates the translation word-by-word, can make local optimization errors, leading to suboptimal outputs.

These problems can result in defective translations, even when the underlying statistical models are theoretically sound. The paper aims to address this by providing a post-processing step to filter out these erroneous translations.

Methods

The core of this paper is the Example-Based Rescoring (EBRS) method. Here's how it works:

  1. Translation Example Retrieval: Given an input sentence, the system retrieves similar sentence pairs (source-target) from a parallel corpus. These retrieved pairs are called "seed sentences."
  2. SMT Decoding: A standard SMT decoder generates multiple translation candidates for the input sentence, using the seed sentence's target part as a starting point or context.
  3. Rescoring: The key innovation is how these candidates are re-evaluated. Instead of solely relying on the TM-LM scores, the EBRS method incorporates an "example-based translation quality hypothesis." This hypothesis states that if a decoder output is very different from its corresponding seed sentence (meaning many modifications were needed), it's more likely to be an error.
  4. Edit Distance: The paper uses edit distance (measuring insertions, deletions, and substitutions) to quantify the difference between the decoder output and the seed sentence.
  5. Two Rescoring Functions: * Weighted Rescoring (TM-LM-EDw): The original TM-LM score is divided by a factor derived from the edit distance (e.g., exp(scale * ED(sd,d))). Larger edit distances lead to lower revised scores.
  6. Probabilistic Rescoring (TM-LM-EDp): A probability is assigned to each candidate by combining the log TM and LM scores with the scaled negative edit distance. This creates a relative ranking of candidates.

The translation candidate with the highest revised score is then selected as the final output.

Datasets

  • Basic Travel Expression Corpus (BTEC): Used for training and evaluation. It contains 157K Japanese-English sentence pairs, with average sentence lengths of 7.7 words (Japanese) and 5.5 words (English). The corpus was split into training (155K), tuning (10K), and evaluation (10K) sets.
  • Test Set: 510 sentences randomly selected from the BTEC for evaluation.
  • Human Reference Translations: Up to 16 human references were used for automatic scoring measures.

Key Insights and Ideas

  • Bridging SMT and EBMT: The paper effectively combines the strengths of Statistical Machine Translation (SMT) and Example-Based Machine Translation (EBMT). SMT provides robust statistical models, while EBMT's reliance on examples offers a different perspective on translation quality.
  • Edit Distance as a Quality Indicator: Using edit distance between the decoder output and a similar "seed" translation example proves to be a powerful heuristic for identifying potentially flawed SMT outputs.
  • Rescoring for Robustness: The proposed rescoring mechanism doesn't just select the highest TM-LM score; it actively penalizes translations that required significant "effort" (modifications) to derive from a similar example, thereby filtering out errors.
  • MT-based Retrieval Advantage: Retrieving seed sentences using outputs from multiple MT systems (even those with varying quality) and then applying the EBRS method yielded the most significant improvements, suggesting that diverse starting points can be beneficial.

Key Results

  • Significant Accuracy Improvement: The EBRS method, particularly the TM-LM-EDw function, achieved a translation accuracy of 92.7%, an improvement of 11.9% over the baseline SMT system (TM-LM).
  • Reduced Word Error Rate (WER): The TM-LM-EDw rescoring reduced WER from 0.255 to 0.010 when applied to already accurate reference translations (simulating optimal retrieval), demonstrating its ability to avoid downgrading good translations.
  • Improved Subjective Scores: Human assessments (ACC scores) also showed substantial gains, with the TM-LM-EDw method achieving an A+B+C score of 0.927 in the MT-based retrieval scenario, compared to 0.796 for the baseline.
  • Outperforming Shortcut Methods: The proposed rescoring method outperformed a "shortcut" method (which skips decoding for exact matches) by 2.9% in translation accuracy, as it can identify and reject incorrectly modified translations.

Limitations and Future Work

  • Black-Box Decoder: The authors acknowledge that they treated the SMT decoder as a "black-box." Future work could involve separating modeling errors from search errors to gain deeper insights.
  • Integration with Decoding: The current method is a post-processing step. Integrating the rescoring logic directly into the decoding process is a promising avenue for further research.
  • Advanced SMT Approaches: Investigating the EBRS method with more advanced SMT techniques (like phrase-based SMT) and different search strategies is suggested.
  • Language Independence: The rescoring function is noted as being language-independent, suggesting its applicability to other language pairs.

Summary updated

BibTeX
@inproceedings{paul2004example,
  author = {Michael Paul and Eiichiro Sumita and Seiichi Yamamoto},
  title = {{Example-based Rescoring of Statistical Machine Translation Output}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure