How Grammatical Features Impact Machine Translation: A New Test Suite for Chinese-English MT Evaluation

Huacheng Song, Yi Li, Yiwen Wu, Yu Liu, Jingxia Lin, Hongzhi Xu2024 · DOI

Abstract

Machine translation (MT) evaluation has evolved toward a trend of fine-grained granularity, enabling a more precise diagnosis of hidden flaws and weaknesses of MT systems from various perspectives. This paper examines how MT systems are potentially affected by certain grammatical features, offering insights into the challenges these features pose and suggesting possible directions for improvement. We develop a new test suite by extracting 7,848 sentences from a multi-domain Chinese-English parallel corpus. All the Chinese text was further annotated with 43 grammatical features using a semi-automatic method. This test suite was subsequently used to evaluate eight state-of-the-art MT systems according to six different automatic evaluation metrics. The results reveal intriguing patterns of MT performance associated with different domains and various grammatical features, highlighting the test suite's effectiveness. The test suite was made publicly available and it will serve as an important benchmark for evaluating and diagnosing Chinese-English MT systems.

Introduction

Evaluating machine translation (MT) quality has traditionally relied on aggregate metrics like BLEU. However, as Neural MT (NMT) and Large Language Models (LLMs) achieve high average scores, these metrics often obscure systematic failures in specific linguistic structures. For Chinese, an isolating language that relies on unique markers rather than inflections, standard evaluation tools often fall short. This research introduces a fine-grained diagnostic test suite designed to pinpoint exactly where and why Chinese-to-English translation breaks down by focusing on 43 specific grammatical features.

Methodology

The researchers developed a test suite of 7,848 sentences sourced from the UM Parallel Corpus, covering seven domains: Education, Laws, News, Science, Spoken, Subtitles, and Thesis. Using a semi-automatic pipeline, they identified and manually verified 43 target grammatical features.

Eight state-of-the-art systems were evaluated:

  • Commercial NMT: Baidu, Niu, Google, and DeepL.
  • Advanced LLMs: Ernie-4, Qwen-turbo, GPT-4o, and Claude-3 Opus.

Performance was measured using six metrics, with a focus on modern neural-based metrics like XCOMET, which provide a more accurate picture of semantic quality than traditional n-gram overlap methods.

Results and Analysis

Overall Performance across Domains

The evaluation revealed that translation quality is highly sensitive to the domain of the text. Systems generally performed best on "Spoken" and "Subtitles" data, which typically feature shorter, simpler sentences. Conversely, performance dipped significantly in the "Thesis" and "Laws" domains, characterized by long, terminology-heavy sentences.

Performance of eight MT systems across seven domains, showing X-AVERAGE scores.

As shown in the data, Google Translate and GPT-4o emerged as the top overall performers. Interestingly, traditional NMT systems like Google and Baidu still maintain a slight edge or parity with the most advanced LLMs in several domains.

Impact of Specific Grammatical Features

The core of the study involved analyzing how specific grammatical markers impact translation accuracy. The researchers identified a clear hierarchy of difficulty among the 43 features.

MT system performance across 43 specific Chinese grammatical features, ranked by average score.
  • The "Hard" Features: Path Preposition Phrases (PathPP) and Relative Constructions (Rel) were the most challenging. For instance, models frequently misinterpreted the preposition yan ("along"), altering the spatial meaning of sentences. Relative constructions often led to confusion between subjects and objects when the marker de was used in complex ways.
  • The "Easy" Features: Features like Progressive ZAI (ZAIProg) and Reciprocal markers (Recp) were handled with high accuracy across all models.

Conclusion

This new test suite provides a vital diagnostic tool for the MT community, moving beyond aggregate scores to reveal the linguistic nuances where even the best models struggle. While LLMs like GPT-4o are closing the gap with specialized NMT systems, specific grammatical structures remain significant hurdles. Future work will focus on expanding feature coverage and incorporating human gold standards to further refine MT diagnostics.

Summary updated

Abstract

Machine translation (MT) evaluation has evolved toward a trend of fine-grained granularity, enabling a more precise diagnosis of hidden flaws and weaknesses of MT systems from various perspectives.This paper examines how MT systems are potentially affected by certain grammatical features, offering insights into the challenges these features pose and suggesting possible directions for improvement.We develop a new test suite by extracting 7,848 sentences from a multi-domain Chinese-English parallel corpus.All the Chinese text was further annotated with 43 grammatical features using a semi-automatic method.This test suite was subsequently used to evaluate eight state-of-the-art MT systems according to six different automatic evaluation metrics.The results reveal intriguing patterns of MT performance associated with different domains and various grammatical features, highlighting the test suite's effectiveness.The test suite was made publicly available and it will serve as an important benchmark for evaluating and diagnosing Chinese-English MT systems.
BibTeX
@misc{song2024how,
  author = {Huacheng Song and Yi Li and Yiwen Wu and Yu Liu and Jingxia Lin and Hongzhi Xu},
  title = {{How Grammatical Features Impact Machine Translation: A New Test Suite for Chinese-English MT Evaluation}},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.117},
}
Figure