Testsuite on Czech–English Grammatical Contrasts

Silvie Cinková, Ondřej BojarEMNLP 2018

TL;DR

This paper introduces a novel testsuite designed to evaluate machine translation (MT) systems on specific grammatical contrasts between Czech and English, focusing on phenomena that lack direct structural equivalents. The study reveals significant differences in how various MT systems handle these challenging linguistic features, highlighting areas for future improvement in cross-lingual MT.

What Problem Does This Paper Address?

Machine translation systems often struggle with languages that have fundamentally different grammatical structures. Czech and English, for instance, exhibit significant typological differences. A prime example is the lack of a direct equivalent for English gerundial clauses and infinitive clauses in modern Czech. While older Czech had a form that could handle these, it's now archaic. Modern Czech relies on more complex structures like finite clauses or nominalizations, creating a "1:n correspondence" problem for MT. This means a single English grammatical structure can map to multiple, context-dependent Czech structures. The authors aim to quantify how well current MT systems handle these specific, challenging grammatical contrasts, hypothesizing that systems mastering these phenomena will perform better overall.

Methods

The core of this research is a meticulously crafted testsuite. For each selected grammatical phenomenon, the authors designed a specific test to check if a candidate translation accurately reflects the phenomenon. This isn't a strict MT evaluation (as there can be multiple correct translations), but rather an analysis of how consistently systems handle these specific structures.

The methodology involved:

  • Data Sourcing: Using a manually parsed, word-aligned parallel treebank of English news texts and their human Czech translations from the Prague Czech-English Dependency Treebank 2.0 (PCEDT 2.0).
  • Phenomenon Selection: Identifying challenging grammatical contrasts based on the authors' experience as language learners, translators, and MT developers. The focus was on English gerundial clauses and infinitive clauses and their Czech equivalents.
  • Test Design: For each phenomenon, they created tests to verify if the Czech translation used the expected structure (e.g., a finite clause for an English gerund). This involved analyzing both the source English sentence and the target Czech translation, leveraging the rich annotation of the PCEDT 2.0.
  • System Evaluation: Applying the testsuite to several MT systems participating in the WMT18 News Translation Task, including state-of-the-art Transformer models, RNN ensembles, and baseline phrase-based systems.

Datasets

The primary dataset used is the Prague Czech-English Dependency Treebank 2.0 (PCEDT 2.0). This is a multi-layered parallel treebank that includes automatic word alignment and manual annotation of syntactic structures (both surface and deep). The sentences for the testsuite were automatically selected from this corpus. The authors also mention using data from the WMT18 News Translation Task for evaluating participating MT systems.

Key Insights and Ideas

  • The "1:n Correspondence" is a Major Hurdle: The study strongly suggests that grammatical structures with no direct one-to-one mapping between languages (like English gerunds to Czech finite clauses) are particularly difficult for MT systems.
  • Reference Translations Aren't Always "Creative": The reference translations tend to use the most typical Czech equivalents, which can sometimes differ from what MT systems produce. This highlights that "best" translation can be subjective and context-dependent.
  • Systemic Differences Emerge: Different MT architectures and training strategies lead to distinct ways of handling these grammatical contrasts. Some systems are more "creative" and deviate more from the reference, while others stick closer.
  • The Value of Linguistic Insight in Evaluation: This work demonstrates the power of linguistically motivated testsuites. Instead of relying solely on general MT metrics, focusing on specific linguistic challenges provides deeper insights into MT system capabilities and weaknesses.

Key Results

The paper presents detailed results in tables and figures, showing the performance of various MT systems on specific phenomena. For instance:

  • EN-gerund-CS-finclause: The reference translation used the expected finite clause structure only in 68.9% of cases, indicating that even human translators don't always opt for the most straightforward structure.
  • System Performance Variation: CUNI Transformer and UEDIN, the top-performing systems overall, showed considerable differences in how they handled the tested phenomena. UEDIN appeared to stay closer to the reference, while CUNI Transformer was more "creative."
  • CUNI Moses Lagging: The phrase-based CUNI Moses system consistently performed worse than more modern architectures on these specific linguistic challenges.
  • "Unknown" Rate: The reference translation sometimes had a higher "Unknown" rate (likely due to word alignment issues), suggesting it might be less verbatim and thus harder to align directly.

Limitations and Future Work

  • Limitations:
    • The testsuite is a "pilot study" and focuses on a limited set of phenomena.
  • The analysis is not a strict MT evaluation, as it anticipates only one correct translation, while multiple might be acceptable.
  • The reliance on automatic parsing and alignment in PCEDT 2.0 might introduce random errors.
  • Future Work:
    • Obtain and compare manual evaluations of individual sentences to validate the automatic findings.
  • Expand the testsuite to cover more linguistic phenomena.
  • Investigate the correlation between the quality rating of translations and their agreement with the reference.
  • Explore how different parsing approaches might improve MT system performance on these contrasts.

Summary updated

BibTeX
@inproceedings{cinkova2018testsuite,
  author = {Silvie Cinková and Ondřej Bojar},
  title = {{Testsuite on Czech–English Grammatical Contrasts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}

Loading related content...

Figure