Leveraging Mandarin as a Pivot Language for Low-Resource Machine Translation between Cantonese and English

King Yiu Suen, Rudolf Chow, Albert Y.s. LamACL 2024

TL;DR

This paper tackles the challenge of low-resource machine translation for Cantonese by using Mandarin as a pivot language. They propose a novel transfer learning approach that leverages pre-trained BART models to initialize auxiliary translation models, which then initialize the final Cantonese-English translation model, achieving state-of-the-art results.

What Problem Does This Paper Address?

Machine translation (MT) for Cantonese has been significantly underdeveloped compared to high-resource languages like Mandarin. This is primarily due to a severe lack of parallel corpora, which are essential for training modern neural machine translation (NMT) models. While Cantonese and Mandarin share linguistic similarities, they are mutually unintelligible, making direct translation challenging. Existing pivot-based MT approaches often involve a two-step decoding process, leading to increased latency and error propagation. This paper aims to bridge this gap by demonstrating how Mandarin, a high-resource language, can effectively serve as a bridge for translating between Cantonese and English, overcoming the limitations of direct translation and naive pivot methods.

Methods

The core of the proposed method is a pivot-based transfer learning strategy. Instead of training NMT models from scratch, the authors leverage pre-trained Bidirectional and Auto-Regressive Transformer (BART) models. The process involves:

  1. Pre-training BART Models: They pre-train Cantonese (Yue), Mandarin (Zh), and English (En) BART models using monolingual corpora. Notably, a Yue BART model is created by continuing the pre-training of a Zh BART model with additional Cantonese monolingual data, exploiting the shared Chinese character system.
  2. Initializing Auxiliary Models: Auxiliary source-pivot (Yue-Zh) and pivot-target (Zh-En) NMT models are initialized. The Yue-Zh model uses the encoder from the Yue BART and the decoder from the Zh BART. Similarly, the Zh-En model uses the encoder from the Zh BART and the decoder from the En BART. These auxiliary models are then fine-tuned on their respective parallel corpora.
  3. Initializing the Source-Target Model: The final Yue-En NMT model is initialized by taking the encoder from the trained Yue-Zh auxiliary model and the decoder from the trained Zh-En auxiliary model. This source-target model is then fine-tuned on Yue-En parallel data.

This approach differs from naive pivot translation by directly training a single source-target model, mitigating error propagation. It also differs from simpler transfer learning by using auxiliary pivot models to guide the initialization, rather than directly transferring from general-purpose BART models.

Datasets

The paper utilizes the following datasets:

  • Monolingual Data:
    • Yue (Cantonese): 10.0 million sentences scraped from Hong Kong online forums (LIHKG, Baby Kingdom, HKEPC).
  • Parallel Data:
    • Zh-En (Mandarin-English): News Commentary v18.1 (WMT24), UN Parallel Corpus v1.0, and WikiMatrix.
  • Yue-Zh (Cantonese-Mandarin): Story books, language learning websites, TED talks, a linguistic study, and a dictionary.
  • Yue-En (Cantonese-English): A language learning website and two dictionaries.

Key Insights and Ideas

  • Mandarin as an Effective Pivot: The study provides strong empirical evidence that Mandarin can be a highly effective pivot language for Cantonese-English translation, despite the linguistic differences.
  • Leveraging Pre-trained BART is Crucial: Transferring parameters from pre-trained BART models significantly boosts performance, especially for the low-resource Cantonese language. Reusing Mandarin resources without additional pre-training for Cantonese is shown to be suboptimal.
  • Auxiliary Model Initialization Benefits: Initializing the final source-target model using encoders and decoders from fine-tuned auxiliary pivot models (Yue-Zh and Zh-En) yields superior results compared to other initialization strategies.
  • Direct Source-Target Training Mitigates Errors: By training a single source-target model initialized from auxiliary models, the paper successfully addresses the error propagation issue inherent in naive two-step pivot translation.
  • Cantonese-Specific Nuances: The qualitative analysis highlights the model's ability to better understand and generate Cantonese idioms and phrases compared to commercial systems, suggesting a deeper linguistic grasp.

Key Results

The proposed pivot-based transfer learning method achieved state-of-the-art results on Cantonese-English translation.

  • Cantonese to English (Yue-En): The highest BLEU score of 19.64 was achieved by initializing the Yue-En model with both the Yue-Zh encoder and the Zh-En decoder. This significantly outperformed random initialization (6.03), naive pivot translation (10.12), and commercial systems like Azure AI Translator (17.50) and Baidu Fanyi (17.21).
  • English to Cantonese (En-Yue): The model also performed exceptionally well in the reverse direction, achieving a BLEU score of 28.22 using an En-Zh encoder and a Zh-Yue decoder. This again surpassed commercial systems (Azure AI Translator: 15.70, Baidu Fanyi: 17.91).
  • Qualitative Improvements: Examples demonstrate the model's superior handling of Cantonese idioms and its ability to produce more natural-sounding translations compared to commercial alternatives.

Limitations and Future Work

Limitations:

  • Monolingual Data Requirement: The method relies on having sufficient monolingual data for the low-resource language (Cantonese) to effectively pre-train the initial BART model.
  • Pivot Language Characteristics: The effectiveness of the approach is contingent on the pivot language (Mandarin) being similar to the low-resource language and having abundant parallel data with the target language (English). This might limit generalization to other low-resource language pairs that don't meet these criteria.

Future Work:

  • Synthetic Data Generation: The authors suggest exploring the generation of synthetic Yue-En parallel data by translating Mandarin sentences from existing Zh-En parallel data into Cantonese. This could further augment the limited parallel data available for Cantonese.

Summary updated

BibTeX
@inproceedings{suen2024leveraging,
  author = {King Yiu Suen and Rudolf Chow and Albert Y.S. Lam},
  title = {{Leveraging Mandarin as a Pivot Language for Low-Resource Machine Translation between Cantonese and English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure