Mixed Embedding of XLM for Unsupervised Cantonese-Chinese Neural Machine Translation (Student Abstract)

Ka Ming Wong, Richard Tzong-Han TsaiAAAI 2022 · DOI

Introduction

Translating between Cantonese (yue) and Standard Chinese (zh) presents a unique challenge in Natural Language Processing. Despite sharing a common writing system, the two languages differ significantly in grammar, syntax, and vocabulary. The primary obstacle for modern translation models is the "low-resource" nature of Cantonese; parallel datasets—the paired sentences required for supervised learning—are exceptionally rare.

To address this, this research introduces a modified Cross-lingual Language Model (XLM) designed for Unsupervised Neural Machine Translation (UNMT). This approach allows the model to learn translation patterns using only monolingual data, bypassing the need for a parallel corpus.

Methodology: Mixed Token Embeddings and Layer-to-Layer Attention

The core innovation of this work lies in two primary modifications to the standard XLM architecture:

  1. Mixed Token Embeddings: Standard models convert words into single vector embeddings. This paper proposes splitting these into Shared Embeddings (trained on both languages to capture commonalities) and Private Embeddings (trained specifically on one language to capture unique dialectal nuances). These are concatenated to form the final input.
  2. Layer-to-Layer Attention: Instead of relying solely on the final encoder layer, the authors implemented a mechanism where each encoder layer coordinates with its corresponding decoder layer, improving structural consistency.
Architecture of the proposed XLM model featuring mixed token embeddings (shared and private) and language-specific embeddings.

The model is first pre-trained as a Masked Language Model (MLM) using Wikipedia data before undergoing the UNMT training phase.

Experimental Setup

The model was trained using monolingual Wikipedia entries for both Cantonese and Standard Chinese. Evaluation was performed on a testing set of 2,000 human-translated parallel sentences. Performance was measured using the BLEU score, comparing the proposed method against a "Non-Translated" baseline and a standard UNMT "Baseline" model.

Results and Key Insights

The "Shared vs. Private" embedding strategy acknowledges that while Cantonese and Chinese are highly related, they are not equivalent. By allowing the model to learn language-specific nuances while leveraging shared structures, the proposed method achieved superior performance.

Comparison of BLEU scores across different models for Cantonese-to-Chinese (yue-zh) and Chinese-to-Cantonese (zh-yue) translation.

As shown in the results table, the proposed method outperformed the baseline in both directions:

  • Cantonese to Chinese (yue-zh): Reached a BLEU score of 69.592, an improvement of 1.088 points over the baseline.
  • Chinese to Cantonese (zh-yue): Reached a BLEU score of 70.99, an improvement of 0.394 points.

In the context of unsupervised machine translation, these gains represent a significant step toward more fluent, human-like output without any initial paired examples.

Conclusion and Future Work

While the results are promising, the authors identify several areas for improvement. The current model is limited by the formal nature of Wikipedia data, which may not capture the informal "slang" prevalent in spoken Cantonese. To address this, the researchers have deployed a functional web service that includes a feedback mechanism, allowing human corrections to be used for future fine-tuning. Future work will focus on developing more Cantonese-specific NLP tasks to further refine the model's linguistic understanding.

Summary updated

BibTeX
@inproceedings{wong2022mixed,
  author = {Ka Ming Wong and Richard Tzong-Han Tsai},
  title = {{Mixed Embedding of XLM for Unsupervised Cantonese-Chinese Neural Machine Translation (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21677},
}
Figure