Statistical Machine Translation of Texts with Misspelled Words

Nicola Bertoldi, Mauro Cettolo, Marcello FedericoNAACL 2010

TL;DR

This paper tackles the challenge of translating text with misspellings, common in user-generated content like social media. They propose an enhanced statistical machine translation (SMT) system that leverages confusion networks to represent and handle spelling variations, significantly improving translation quality on noisy input.

What Problem Does This Paper Address?

The explosion of user-generated content online, from blogs and social media to reviews, has created a massive amount of multilingual text. However, this content is often rife with informal language, abbreviations, and crucially, misspellings. Traditional SMT systems, trained on clean, formal text, struggle significantly when faced with these "noisy" inputs, leading to degraded translation quality. This paper aims to bridge this gap by investigating the impact of misspellings on SMT performance and proposing a method to mitigate these errors.

Methods

The core of the proposed solution lies in an enhanced SMT system that preprocesses the input text to handle misspellings before translation. This involves:

  1. Spelling Variation Generation: The system assumes misspellings are primarily due to typing errors (substitutions, insertions, deletions, transpositions). It models these potential errors at a character level, assigning probabilities to character alternatives based on keyboard layout distances.
  2. Confusion Network (CN) Construction: These character-level variations are then used to build a character-based Confusion Network. This network represents multiple possible corrected spellings for each word.
  3. Word-Based CN Conversion: The character-based CN is then transformed into a word-based CN. This compact representation efficiently stores a large number of spelling variations.
  4. SMT Decoding with CNs: Finally, the SMT decoder (specifically, the Moses decoder) is adapted to translate these word-based CNs, effectively considering multiple spelling hypotheses for each input word. This is a departure from traditional SMT which typically processes a single input string or a limited set of n-best lists.

Datasets

The experiments were conducted on two main datasets:

  • Europarl: A large, standard dataset for machine translation, used for English-to-Italian translation.
  • Weather Forecast (WF) bulletins: A smaller, but high-quality dataset, used for both English-to-Italian and Italian-to-English translation.

The authors also created corrupted versions of these test sets by introducing artificial noise (non-word, real-word, and random errors) to evaluate their system's robustness. The module for generating spelling variations was trained on additional large corpora (4M and 16M words) in English and Italian.

Key Insights and Ideas

  • Misspellings are a significant bottleneck for SMT: Even low levels of noise can drastically reduce translation quality, with performance degradation observed even at 1% noise.
  • Noise source matters for recovery, not impact: While different types of misspellings (non-word, real-word, random) have a similar negative impact on SMT performance, the ability to recover from them varies. Real-word errors (e.g., "their" instead of "there") are significantly harder to correct than random or non-word errors.
  • Leveraging multiple spelling variations is key: For random and non-word errors, a single spelling alternative can largely restore translation quality. However, for real-word errors, multiple variations are crucial for effective recovery.
  • Over-correction can be detrimental: At very low noise levels, the error recovery module can sometimes introduce more errors than it fixes, leading to a slight performance drop. This highlights the need for intelligent activation strategies.

Key Results

  • The proposed error-recovering SMT system significantly outperforms a standard SMT system on noisy input across various noise levels and types.
  • For random and non-word noise, the system can fully recover translation performance up to 10% noise levels, and performance degradation is limited to around 5% even at 20% noise.
  • Real-word errors are more challenging; recovery is optimal for noise levels up to 2%, with a slight degradation thereafter.
  • The system's effectiveness is demonstrated on both the large Europarl task and the smaller WF task, showing good portability.

Limitations and Future Work

  • Computational Cost: While the CN decoding is efficient, the process of generating spelling variations (Steps 1-4) is currently computationally expensive, partly due to a naive implementation. Future work will focus on optimizing this process.
  • Activation Strategy: The current system can introduce spurious errors at low noise levels. Future research will explore effective strategies for activating the error recovery module, possibly by estimating the noise rate of the input text.
  • Alternative Variation Generation: The current method relies on character-level error modeling. Investigating other approaches, such as using n-grams of words for variation generation, is suggested.

Summary updated

BibTeX
@inproceedings{bertoldi2010statistical,
  author = {Nicola Bertoldi and Mauro Cettolo and Marcello Federico},
  title = {{Statistical Machine Translation of Texts with Misspelled Words}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure