Target-Bidirectional Neural Models for Machine Transliteration

Andrew M. Finch, Lemao Liu, Xiaolin Wang, Eiichiro SumitaNEWS@ACM 2016 · OpenReview

TL;DR

This paper introduces a novel end-to-end neural network system for machine transliteration that significantly outperforms previous phrase-based statistical methods. By employing a "target-bidirectional" approach with ensembles of neural networks, the system achieves state-of-the-art results on many language pairs, demonstrating the power of neural models for this complex task.

What Problem Does This Paper Address?

Machine transliteration is the task of converting words from one script to another, aiming to preserve pronunciation. For example, converting "Tokyo" (Latin script) to "東京" (Japanese Kanji). Historically, phrase-based statistical machine translation (PB-SMT) techniques have been dominant. However, these methods have limitations, particularly in how errors can accumulate during the generation process, leading to unbalanced output with good prefixes but degraded suffixes. Previous neural approaches, while promising, often underperformed PB-SMT in isolation. This paper tackles the challenge of developing a purely neural end-to-end system that can match or surpass the performance of established PB-SMT systems.

Methods

The core innovation is the target-bidirectional agreement model. Unlike traditional sequence-to-sequence models that generate output in one direction (e.g., left-to-right), this approach uses two recurrent neural networks (RNNs), specifically LSTMs, that generate target sequences from both left-to-right and right-to-left. These two "bidirectional" outputs are then combined to encourage agreement, resulting in more balanced and higher-quality transliterations.

Additionally, the authors leverage ensembles of neural networks. Instead of relying on a single trained model, they combine the predictions of multiple networks. These networks are selected from different training runs and different epochs of training, based on their performance on development data. This ensemble approach further boosts performance and robustness.

The system is designed to be language-independent, meaning no specific linguistic knowledge is hardcoded for different language pairs. The only preprocessing is uppercasing English to ensure consistency.

Datasets

The paper evaluates the system on various language pairs for the NEWS (Named Entity With Script) shared evaluation. Specific datasets mentioned include:

  • English to Bengali (EnBa)
  • Chinese to English (ChEn)
  • English to Chinese (EnCh)
  • English to Hebrew (EnHe)
  • English to Hindi (EnHi)
  • English to Japanese Katakana (EnJa)
  • English to Kannada (EnKa)
  • English to Korean Hangul (EnKo)
  • English to Persian (EnPe)
  • English to Tamil (EnTa)
  • English to Thai (EnTh)
  • English to Japanese Kanji (JnJk)
  • Thai to English (ThEn)

The Arabic-English dataset is also mentioned, though the system did not perform well on it.

Key Insights and Ideas

  • Bidirectionality is Key for Balance: The target-bidirectional approach is crucial for overcoming the "prefix-good, suffix-bad" problem inherent in unidirectional sequence-to-sequence models. By generating from both ends, the system achieves a more consistent quality throughout the transliterated word.
  • Ensembles Smooth Out Variance: Combining multiple neural networks, especially those trained differently (different runs, different epochs), significantly improves performance and robustness. This is particularly evident on language pairs like English-Japanese (En-Ja), which showed high variance in single-model performance.
  • Neural Networks Can Surpass PB-SMT: The results demonstrate that purely neural end-to-end systems can achieve state-of-the-art performance, challenging the long-held dominance of PB-SMT in this domain.
  • Generalization vs. Memorization: The authors hypothesize that for tasks like Arabic-English transliteration, where short vowels are not explicitly written but must be inferred, phrase-based systems that excel at memorizing training data might outperform neural networks that are geared more towards generalization.

Key Results

The paper reports significant improvements over previous systems. For instance:

  • On English to Hindi (EnHi), the 12-ensemble system achieved an accuracy of 0.709, surpassing the previous system's 0.696 and the baseline's 0.270.
  • On English to Japanese Katakana (EnJa), the 12-ensemble system reached 0.464, a notable jump from the previous system's 0.407 and the baseline's 0.209.
  • The system matched or surpassed previous state-of-the-art on most language pairs.
  • However, the system did not match previous performance on English-Japanese Kanji, English-Hangul, and Arabic-English.

Limitations and Future Work

  • Arabic-English Challenge: The system struggled with Arabic-English transliteration, which the authors attribute to the need for inferring unwritten short vowels, a task where memorization-heavy PB-SMT might have an advantage. Further research is needed to understand and address this.
  • Large Target Vocabularies: For tasks with very large target vocabularies (e.g., English-Japanese Kanji, English-Hangul), the neural system's performance was not as strong. This suggests potential limitations for neural models in such scenarios.
  • Architecture Optimization: The authors acknowledge that the chosen network architecture was fixed due to computational constraints, and it's possible that better architectures exist. Future work could involve exploring more optimal architectures.
  • Generalization vs. Memorization Trade-off: Understanding and balancing generalization and memorization in neural models for specific transliteration challenges remains an open area for future research.

Summary updated

BibTeX
@inproceedings{finch2016target,
  author = {Andrew M. Finch and Lemao Liu and Xiaolin Wang and Eiichiro Sumita},
  title = {{Target-Bidirectional Neural Models for Machine Transliteration}},
  booktitle = {NEWS@ACM},
  year = {2016},
  url = {https://openreview.net/forum?id=OkzQahpULQ-},
}
Figure