Robust Neural Machine Translation with Joint Textual and Phonetic Embedding

Hairong Liu, Mingbo Ma, Liang Huang, Hao Xiong, Zhongjun He

Abstract

Neural machine translation (NMT) is notoriously sensitive to noises, but noises are almost inevitable in practice. One special kind of noise is the homophone noise, where words are replaced by other words with similar pronunciations. We propose to improve the robustness of NMT to homophone noises by 1) jointly embedding both textual and phonetic information of source sentences, and 2) augmenting the training dataset with homophone noises. Interestingly, to achieve better translation quality and more robustness, we found that most (though not all) weights should be put on the phonetic rather than textual information. Experiments show that our method not only significantly improves the robustness of NMT to homophone noises, but also surprisingly improves the translation quality on some clean test sets.

Introduction

Modern Neural Machine Translation (NMT) models, such as the Transformer, have achieved state-of-the-art performance but remain remarkably brittle when faced with input perturbations. A particularly prevalent form of noise in real-world applications is "homophone noise," where a correct word is replaced by a phonetically similar but semantically distinct token. This occurs frequently in pipelines involving Automatic Speech Recognition (ASR) or phonetic input methods (e.g., Pinyin for Mandarin). Standard NMT models, which rely on discrete textual tokens, struggle with these swaps as they fundamentally alter the input embeddings, leading to catastrophic translation failures.

Methodology

To address this vulnerability, we propose a joint embedding layer that incorporates both textual and phonetic information.

Joint Textual and Phonetic Embedding

For each input word, the model retrieves its corresponding phonetic units (e.g., Pinyin syllables for Mandarin). These phonetic units are embedded and averaged into a single vector. This phonetic vector is then combined with the standard textual word embedding using a weighted sum controlled by a hyperparameter β:

Combined_Embedding = (1 - β) * Text_Embedding + β * Phonetic_Embedding

This allows the model to leverage phonetic similarity to bridge the gap between a word and its homophone.

Data Augmentation

In addition to the architectural change, we employ a data augmentation strategy. During training, words in the source sentences are randomly replaced with their homophones. This teaches the model to explicitly handle phonetic variations and reinforces the robustness provided by the joint embeddings.

Experimental Setup

We evaluated our approach on a Mandarin-to-English translation task using an extended NIST corpus (~2M sentence pairs). Robustness was tested using two synthetic noisy benchmarks derived from the NIST06 set: NoisySet1 (10% homophone replacement) and NoisySet2 (20% replacement). We compared the baseline Transformer against models with varying β values.

Results and Discussion

Our experiments reveal that incorporating phonetic information significantly enhances model performance in both clean and noisy scenarios.

The Power of Phonetics

A key finding is that the model performs best when it relies heavily on phonetic information. As shown in the figure below, the baseline model's BLEU score drops sharply as noise increases. In contrast, models with higher β values exhibit much greater stability.

BLEU scores across different noise levels and β values

The optimal balance was found at β = 0.95, where the model utilizes 95% phonetic and 5% textual information. On the clean NIST06 dev set, the baseline Transformer achieved a BLEU score of 45.97, while the β = 0.95 model jumped to 48.91—a massive ~3-point improvement. When subjected to 20% homophone noise (NoisySet2), the baseline model's BLEU plummeted to 37.11, while the β = 0.95 model only dropped to 42.66. When combined with homophone data augmentation, the model maintained a BLEU of 46.47, effectively neutralizing the impact of the noise.

Accidental Regularization

We hypothesize that phonetic embeddings act as a powerful regularizer. By mapping words with similar sounds to nearby points in the embedding space, the model learns a smoother representation. This is supported by t-SNE visualizations where syllables with similar pronunciations naturally cluster together.

t-SNE visualization of phonetic embeddings showing clustering of similar sounds

This clustering helps the model generalize better even on clean data, effectively neutralizing the impact of minor variations in the input and improving overall translation quality.

Conclusion

By jointly embedding textual and phonetic information, we significantly improve the robustness of NMT models to homophone noise. Our findings highlight the importance of phonetic cues, showing that a 95% phonetic weighting not only protects against noise but also enhances general translation quality. Future work includes incorporating tone information for better disambiguation and exploring the theoretical underpinnings of the observed regularization effect.

Summary updated

Abstract

Neural machine translation (NMT) is notoriously sensitive to noises, but noises are almost inevitable in practice. One special kind of noise is the homophone noise, where words are replaced by other words with similar pronunciations. We propose to improve the robustness of NMT to homophone noises by 1) jointly embedding both textual and phonetic information of source sentences, and 2) augmenting the training dataset with homophone noises. Interestingly, to achieve better translation quality and more robustness, we found that most (though not all) weights should be put on the phonetic rather than textual information. Experiments show that our method not only significantly improves the robustness of NMT to homophone noises, but also surprisingly improves the translation quality on some clean test sets.
BibTeX
@inproceedings{liu2019robust,
  author = {Hairong Liu and Mingbo Ma and Liang Huang and Hao Xiong and Zhongjun He},
  title = {{Robust Neural Machine Translation with Joint Textual and Phonetic Embedding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=-QSVeeJbOmi5},
}
Figure