Robust Neural Machine Translation with Joint Textual and Phonetic Embedding

Hairong Liu, Mingbo Ma, Liang Huang, Hao Xiong, Zhongjun He

Abstract

Neural machine translation (NMT) is notoriously sensitive to noises, but noises are almost inevitable in practice. One special kind of noise is the homophone noise, where words are replaced by other words with similar pronunciations. We propose to improve the robustness of NMT to homophone noises by 1) jointly embedding both textual and phonetic information of source sentences, and 2) augmenting the training dataset with homophone noises. Interestingly, to achieve better translation quality and more robustness, we found that most (though not all) weights should be put on the phonetic rather than textual information. Experiments show that our method not only significantly improves the robustness of NMT to homophone noises, but also surprisingly improves the translation quality on some clean test sets.

Introduction

Standard Neural Machine Translation (NMT) models are notoriously fragile when faced with "homophone noise"—errors where a word is replaced by another that sounds similar but carries a different meaning. Standard models treat unique strings as distinct entities; thus, a typo like "there" for "their" or an Automatic Speech Recognition (ASR) error can result in radically different embeddings. In architectures like the Transformer, global attention allows a single noisy word to propagate errors throughout the sentence, often leading to "hallucinations" or irrelevant outputs. This is especially problematic for languages like Mandarin Chinese, which has a high density of homophones.

Methodology

To address this, we propose two primary strategies:

Joint Textual and Phonetic Embedding

Instead of a single vector for a word $a$, we create a joint representation $\pi_{joint}$ combining textual and phonetic information: $$\pi_{joint} = (1 - \beta) \cdot \pi(a) + \beta \cdot \pi(\psi(a))$$ where $\pi(a)$ is the standard textual embedding and $\pi(\psi(a))$ is the phonetic embedding derived from Pinyin syllables. A key finding is the "Beta Factor": the model performs best when $\beta \approx 0.95$, prioritizing phonetic signals over textual characters.

Data Augmentation

We further harden the model by augmenting the training set, randomly replacing words with homophones (10-20% probability) to create a "noisy" training version.

Experimental Results

We evaluated our approach using the NIST corpus (2 million Mandarin-English pairs) and standard NIST test sets. To measure robustness, we created "NoisySet1" and "NoisySet2" by injecting 10% and 20% homophone noise into the NIST06 dev set.

Performance on Clean Data

The joint embedding model significantly outperformed the baseline Transformer across various test sets. Notably, at $\beta=0.95$, the model achieved a BLEU score of 48.91 on the NIST06 clean set, a ~3 point gain over the baseline.

BLEU scores across various NIST test sets for different beta values.

Robustness to Noise

The baseline model's performance drops sharply under noise. However, the joint embedding approach maintains high quality even in the presence of 20% noise (NoisySet2).

Impact of homophone noise on BLEU scores for different beta configurations.

When combining joint embeddings with data augmentation, the model achieved 46.47 BLEU on NoisySet2, nearly matching the baseline's performance on perfectly clean data.

Discussion

Phonetics as Regularization

Surprisingly, phonetic information improves performance even on clean sets. This suggests phonetic embeddings act as a form of regularization, helping the model generalize by grouping phonetically related concepts.

The "Sweet Spot" for $\beta$

While a purely phonetic model ($\beta = 1.0$) loses semantic nuance, a "mostly" phonetic model ($\beta = 0.95$) provides the best balance. The sound provides the primary anchor, while the text provides the final disambiguating nudge. t-SNE visualizations confirm that the model naturally clusters similar sounds (e.g., zhen and zheng) without explicit instruction.

Conclusion

Our method demonstrates that incorporating phonetic information significantly boosts NMT robustness to homophone noise and improves general translation quality. Future work includes testing on languages with deeper orthographies (like English) and incorporating tone information for better disambiguation in Mandarin.

Summary updated

Abstract

Neural machine translation (NMT) is notoriously sensitive to noises, but noises are almost inevitable in practice. One special kind of noise is the homophone noise, where words are replaced by other words with similar pronunciations. We propose to improve the robustness of NMT to homophone noises by 1) jointly embedding both textual and phonetic information of source sentences, and 2) augmenting the training dataset with homophone noises. Interestingly, to achieve better translation quality and more robustness, we found that most (though not all) weights should be put on the phonetic rather than textual information. Experiments show that our method not only significantly improves the robustness of NMT to homophone noises, but also surprisingly improves the translation quality on some clean test sets.
BibTeX
@inproceedings{liu2019robust,
  author = {Hairong Liu and Mingbo Ma and Liang Huang and Hao Xiong and Zhongjun He},
  title = {{Robust Neural Machine Translation with Joint Textual and Phonetic Embedding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=-QSVeeJbOmi5},
}
Figure