Machine Translation Evaluation Benchmark for Wu Chinese: Workflow and Analysis

Hongjian Yu, Yiming Shi, Zherui Zhou, Christopher Haberland

TL;DR

This paper introduces the first comprehensive benchmark dataset for Wu Chinese machine translation, addressing its status as a textually under-resourced language. By creating a new FLORES+ dataset and conducting thorough analyses, the authors pave the way for improved MT systems for this widely spoken but digitally marginalized Sinitic language.

What Problem Does This Paper Address?

Wu Chinese, spoken by approximately 83 million people, is the second-largest Sinitic language group in China. Despite its large speaker base, Wu Chinese is considered "textually under-resourced," meaning there's a significant lack of digital text data available for natural language processing (NLP) tasks, particularly machine translation (MT). This scarcity poses a major hurdle for developing effective MT systems that can bridge the gap between Wu Chinese and other languages, especially other Sinitic varieties like Mandarin and Yue. The paper aims to alleviate this by creating a standardized evaluation benchmark and exploring the challenges associated with Wu Chinese NLP.

Methods

The core of the paper's methodology revolves around creating and validating a new Wu Chinese dataset for machine translation.

  • Dataset Creation: The authors built upon the FLORES+ benchmark, which consists of parallel English-to-language data. They manually translated English sentences into Wu Chinese, specifically focusing on the Chongming dialect. This process involved careful consideration of Wu Chinese's unique phonological and orthographic characteristics, including its dual pronunciation system (Wendu and Baidu) and the historical evolution of its writing system.
  • Orthographic Standardization: A significant challenge was the lack of a universally accepted writing system for Wu Chinese. The authors developed a guideline for selecting Hanzi (Chinese characters) based on historical usage, semantic meaning, and frequency in vulgar texts, referencing established Wu Chinese dictionaries.
  • Normalization and Segmentation: To prepare the data for NLP models, they implemented normalization techniques to handle character variations and developed segmentation tools tailored for Wu Chinese, acknowledging that Chinese languages don't use spaces to delimit words.
  • Evaluation: The primary evaluation method involved training language identification models. These models were used to assess the compatibility and consistency of the newly created Wu Chinese dataset with existing Wu Chinese text resources (like Wikimedia) and other Sinitic languages (Mandarin and Yue). This approach helps validate the quality and usability of the benchmark.

Datasets

  • FLORES+ Wu Dataset: This is the primary contribution of the paper. It's a manually translated, open-source dataset of parallel English-Wu Chinese sentences, designed to serve as both a training corpus and an evaluation benchmark for MT models. It contains 997 sentences.
  • Wikimedia Data: Wu, Mandarin, and Yue Wikimedia dumps were used for training and evaluating language identification models.
  • Existing Wu Chinese Dictionaries: The paper references and utilized dictionaries like "简明吴方言词典" (Concise Wu Dialect Dictionary) and "上海话大词典" (Shanghai Dialect Big Dictionary) to aid in orthographic standardization.

Key Insights and Ideas

  • Wu Chinese is a "data-scarce" language: Despite its large speaker population, the lack of readily available digital text makes Wu Chinese a challenging target for modern NLP techniques.
  • Orthographic ambiguity is a major hurdle: The absence of a single, standardized writing system for Wu Chinese necessitates careful selection and normalization of characters, making it distinct from languages with more established orthographies.
  • Dialectal variation complicates standardization: Wu Chinese has significant internal dialectal variations, and the chosen Chongming dialect, while representative of Northern Wu, might not fully capture the nuances of all Wu varieties, particularly Southern Wu dialects.
  • Language identification is a useful proxy for dataset quality: Training language identification models proved effective in validating the internal consistency of the FLORES+ Wu dataset and its compatibility with other Wu text resources.

Key Results

  • High accuracy in language identification: The language identification models trained on the FLORES+ Wu dataset and Wikimedia data demonstrated high accuracy (often above 90%) in distinguishing between Mandarin, Wu, and Yue, validating the dataset's internal consistency and quality.
  • Challenges in Mandarin-Wu distinction: The Mandarin-Wu language identification model showed a tendency to misclassify Mandarin texts as Wu, suggesting that while the Wu dataset helps capture Wu features, it might be less effective at recognizing distinct Mandarin features.
  • Impact of segmentation on performance: The paper highlights that limitations in Wu Chinese segmentation tools can hinder the model's ability to learn accurate representations of linguistic constituents, leading to misclassifications.
  • Influence of Mandarin: The tendency for Wu Chinese to be influenced by Mandarin was observed, with classifiers mislabeling Wu data containing Mandarin-influenced grammar particles.

Limitations and Future Work

  • Limited scope of the Wu dataset: The dataset primarily uses the Chongming dialect, which might not be representative of all Wu Chinese varieties, especially Southern Wu dialects. This could limit the dataset's effectiveness for tasks involving those dialects.
  • Segmentation tool limitations: The current segmentation tools for Wu Chinese are not as robust as those for Mandarin or Yue, impacting the model's ability to accurately represent Wu constituents.
  • Orthographic standardization challenges: While efforts were made to standardize orthography, the inherent ambiguity and variation in Wu Chinese writing present ongoing challenges.
  • Future Work: The authors suggest optimizing the tokenization scheme with better segmentation tools, expanding the auxiliary dictionary for Wu Chinese, and aligning training and testing datasets to cover a broader range of dialects. They also emphasize the need for AI scientists to understand the linguistic properties of their training and testing data for Wu Chinese.

Summary updated

BibTeX
@inproceedings{yu2024machine,
  author = {Hongjian Yu and Yiming Shi and Zherui Zhou and Christopher Haberland},
  title = {{Machine Translation Evaluation Benchmark for Wu Chinese: Workflow and Analysis}},
  booktitle = {WMT},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.47},
}

Loading related content...

Figure