Evaluating Low-Resource Machine Translation between Chinese and Vietnamese with Back-Translation

Hongzheng Li, Heyan HuangarXiv 2020 · arXiv

Abstract

This study investigates the impact of back-translation (BT) on machine translation (MT) for the low-resource Chinese-Vietnamese language pair. By comparing Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) across different tokenization strategies and data scales, the researchers find that while BT generally enhances performance, its effectiveness varies significantly by translation direction and model type. Notably, character-based models consistently outperform word-based ones, and increasing synthetic data volume often leads to diminishing returns, challenging common assumptions in high-resource MT research.

Introduction

Neural Machine Translation (NMT) has revolutionized the field but remains heavily dependent on massive parallel datasets. For "low-resource" language pairs like Chinese and Vietnamese, such data is scarce. Back-translation—a technique where a model translates monolingual target-side data back into the source language to create "synthetic" parallel data—is a standard solution.

However, most BT research focuses on European languages. Chinese and Vietnamese present unique challenges: they belong to different language families (Sino-Tibetan vs. Austroasiatic) and use different scripts, yet they share deep historical ties and similar analytic (SVO) structures. This paper systematically tests if Western-centric BT assumptions hold true for this specific Asian language pair.

Methodology

The researchers conducted a comprehensive comparison across several dimensions:

  • Model Architectures: Traditional Statistical Machine Translation (SMT) using the Moses toolkit was compared against NMT using a 2-layer LSTM (OpenNMT).
  • Tokenization: Both character-based and word-based settings were tested. This is critical for Vietnamese, where spaces separate syllables rather than words, and Chinese, which uses no spaces.
  • Back-Translation Pipeline: The process involved training an intermediate system (e.g., Vietnamese → Chinese) to translate monolingual data into synthetic pairs, which were then combined with authentic data to train the final system.
  • Incremental Scaling: Synthetic data was added in 10k increments (from 10k to 50k) to a base of 50k authentic pairs to identify the optimal data volume.
The back-translation pipeline used to generate synthetic parallel data for Chinese-Vietnamese MT.

Datasets

The study utilized a high-quality, manually processed dataset of 56,610 authentic sentence pairs from Vietnamese news websites. A separate 200k parallel corpus from various domains (textbooks, dialogues, etc.) provided the monolingual data for the back-translation process. Performance was evaluated using BLEU and METEOR metrics.

Key Insights and Results

Character-Based vs. Word-Based Modeling

Character-based models almost always outperformed word-based ones. Character-level modeling likely bypasses errors introduced by imperfect word segmentation tools and better handles the monosyllabic nature of both languages.

The "More is Better" Myth

Unlike high-resource scenarios, adding more synthetic data sometimes decreased performance. In many NMT experiments, scores peaked before reaching the maximum amount of synthetic data, suggesting that too much "noisy" synthetic data can overwhelm a small authentic dataset.

BLEU score performance across varying amounts of synthetic data for SMT and NMT models.

Directional Asymmetry and Model Performance

BT was significantly more effective for the Chinese → Vietnamese direction than the reverse. In the character-based Chinese → Vietnamese task, the baseline BLEU of 14.29 rose to 14.95 with 50k synthetic sentences. For NMT, BLEU rose from 11.68 to 12.34, peaking at 30k synthetic sentences.

Qualitatively, NMT proved superior at handling entity names and complex syntactic structures, such as the Chinese "把" (BA) construction, compared to SMT, which often produced less coherent translations or left names untranslated.

Conclusion

The study highlights that BT is a viable but sensitive technique for Chinese-Vietnamese MT. The results were limited by a relatively small dataset, which precluded the use of the Transformer architecture or Byte Pair Encoding (BPE). Future work will focus on expanding dataset sizes to enable Transformer-based experiments and exploring the combination of back-translation with transfer learning to further boost performance in low-resource settings.

Summary updated

Abstract

Back translation (BT) has been widely used and become one of standard techniques for data augmentation in Neural Machine Translation (NMT), BT has proven to be helpful for improving the performance of translation effectively, especially for low-resource scenarios. While most works related to BT mainly focus on European languages, few of them study languages in other areas around the world. In this paper, we investigate the impacts of BT on Asia language translations between the extremely low-resource Chinese and Vietnamese language pair. We evaluate and compare the effects of different sizes of synthetic data on both NMT and Statistical Machine Translation (SMT) models for Chinese to Vietnamese and Vietnamese to Chinese, with character-based and word-based settings. Some conclusions from previous works are partially confirmed and we also draw some other interesting findings and conclusions, which are beneficial to understand BT further.
BibTeX
@misc{li2020evaluating,
  author = {Hongzheng Li and Heyan Huang},
  title = {{Evaluating Low-Resource Machine Translation between Chinese and Vietnamese with Back-Translation}},
  year = {2020},
  eprint = {2003.02197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.02197},
}
Figure