Introduction
Translating between Cantonese (yue) and Standard Chinese (zh) presents a unique challenge in Natural Language Processing. Despite sharing a common writing system, the two languages differ significantly in grammar, syntax, and vocabulary. The primary obstacle for modern translation models is the "low-resource" nature of Cantonese; parallel datasets—the paired sentences required for supervised learning—are exceptionally rare.
To address this, this research introduces a modified Cross-lingual Language Model (XLM) designed for Unsupervised Neural Machine Translation (UNMT). This approach allows the model to learn translation patterns using only monolingual data, bypassing the need for a parallel corpus.
Methodology: Mixed Token Embeddings and Layer-to-Layer Attention
The core innovation of this work lies in two primary modifications to the standard XLM architecture:
- Mixed Token Embeddings: Standard models convert words into single vector embeddings. This paper proposes splitting these into Shared Embeddings (trained on both languages to capture commonalities) and Private Embeddings (trained specifically on one language to capture unique dialectal nuances). These are concatenated to form the final input.
- Layer-to-Layer Attention: Instead of relying solely on the final encoder layer, the authors implemented a mechanism where each encoder layer coordinates with its corresponding decoder layer, improving structural consistency.
The model is first pre-trained as a Masked Language Model (MLM) using Wikipedia data before undergoing the UNMT training phase.
Experimental Setup
The model was trained using monolingual Wikipedia entries for both Cantonese and Standard Chinese. Evaluation was performed on a testing set of 2,000 human-translated parallel sentences. Performance was measured using the BLEU score, comparing the proposed method against a "Non-Translated" baseline and a standard UNMT "Baseline" model.
Results and Key Insights
The "Shared vs. Private" embedding strategy acknowledges that while Cantonese and Chinese are highly related, they are not equivalent. By allowing the model to learn language-specific nuances while leveraging shared structures, the proposed method achieved superior performance.
As shown in the results table, the proposed method outperformed the baseline in both directions:
- Cantonese to Chinese (yue-zh): Reached a BLEU score of 69.592, an improvement of 1.088 points over the baseline.
- Chinese to Cantonese (zh-yue): Reached a BLEU score of 70.99, an improvement of 0.394 points.
In the context of unsupervised machine translation, these gains represent a significant step toward more fluent, human-like output without any initial paired examples.
Conclusion and Future Work
While the results are promising, the authors identify several areas for improvement. The current model is limited by the formal nature of Wikipedia data, which may not capture the informal "slang" prevalent in spoken Cantonese. To address this, the researchers have deployed a functional web service that includes a feedback mechanism, allowing human corrections to be used for future fine-tuning. Future work will focus on developing more Cantonese-specific NLP tasks to further refine the model's linguistic understanding.
Summary updated