SAO WMT19 Test Suite: Machine Translation of Audit Reports

Tereza Vojtěchová, Michal Novák, Miloš Klouček, Ondřej BojarACL 2019

TL;DR

This paper introduces a new test suite of audit reports for machine translation (MT) evaluation, specifically designed to assess domain adaptation and document-level coherence. Their findings suggest that while general-domain MT systems perform surprisingly well on audit reports, deep domain knowledge is crucial for truly accurate translations, and current automatic metrics are insufficient for capturing these nuances.

What Problem Does This Paper Address?

Machine translation (MT) systems are often trained on vast amounts of general-domain text, like news articles. However, when these systems are applied to specialized domains (e.g., legal, medical, or financial documents), their performance can degrade significantly due to domain mismatch. This paper investigates two key challenges: 1) how well do current, large-scale MT systems generalize to the specialized domain of audit reports, and 2) how well do they handle document-level coherence, which is crucial for understanding complex texts. The authors aim to provide a benchmark for evaluating MT systems on this specific domain and to highlight the limitations of current evaluation methods.

Methods

The core of this paper is the creation and evaluation of a new test suite. The authors collected audit reports from the Supreme Audit Office of the Czech Republic (SAO). These reports were converted from PDF to plain text, and then sentence-aligned for Czech, English, and German to create a tri-parallel test set. They also included a sublease agreement document to evaluate a different type of specialized text.

For evaluation, they employed both automatic metrics (BLEU, chrF3, nCDER, etc.) and a rigorous manual evaluation. The manual evaluation was conducted by SAO employees, who possess domain expertise. They developed a set of criteria focusing on language resources, vocabulary adequacy and clarity, syntax, and coherence. Crucially, they deliberately excluded reference translations during the manual evaluation to avoid biasing annotators and to better assess the inherent quality of the translations.

Datasets

  • SAO WMT19 Test Suite: A newly created dataset consisting of 10 multi-language audit reports from the Supreme Audit Office of the Czech Republic. This suite was used for the WMT19 News Translation Task.
  • Sublease Agreement: An additional document from the domain of agreements, also used for evaluation.

The paper notes that the documents in the SAO test suite were not part of the WMT19 constrained training data, likely due to their PDF format.

Key Insights and Ideas

  • Domain Adaptation is Still Key: Despite the impressive performance of large MT systems on general news, specialized domains like audit reports reveal significant challenges. While systems might produce fluent output, deep factual and domain-specific knowledge is essential for accurate translation.
  • Automatic Metrics Fall Short: The paper strongly argues that standard automatic metrics (like BLEU) are "practically useless" for evaluating the quality of translations in specialized domains, especially when deep factual understanding is required. The variance in scores across individual documents highlights this limitation.
  • Manual Evaluation is Indispensable: For specialized domains, human evaluation by domain experts is critical. The annotators' ability to discern subtle errors in terminology and factual implications far surpasses what automatic metrics can capture.
  • Document-Level Coherence Matters: While MT systems are improving, they still struggle with maintaining consistency and coherence across an entire document, particularly in lexical choices.
  • Reference Translations Can Be Misleading: The authors found that even carefully crafted reference translations can be insufficient or even misleading for evaluating specialized texts, as they might not capture all valid interpretations or nuances.

Key Results

  • General MT Systems Perform Well (Superficially): The study found that current MT systems, even those trained on general news, can achieve surprisingly good automatic scores on audit reports.
  • Manual Evaluation Reveals Deeper Issues: The manual evaluation, however, uncovered significant errors in terminology, grammar, and coherence that were missed by automatic metrics.
  • Annotator Agreement is Challenging: The study noted that even among domain experts, there can be disagreements on translation quality, especially concerning subtle lexical choices.
  • Specific Domain Errors: For audit reports, errors related to "wrong translation of terms" and "grammatical correctness" were common. In the sublease agreement, the most severe flaw was the consistent mistranslation of "tenant" and "lessee" into a single Czech term ("nájemce"), rendering the text incomprehensible.
  • System Performance Varies: The performance of MT systems varied significantly, with some systems (like CUNI-Transformer-T2T-2018) performing better than others (like online-X and online-G) in both automatic and manual evaluations.

Limitations and Future Work

  • Limited Annotator Pool: For the English-German and German-English translations, only a single annotator was available, which limits the ability to measure inter-annotator agreement.
  • Small Sample Size for Manual Evaluation: The manual evaluation was conducted on segments rather than full documents, and the number of annotated segments was relatively small, making it difficult to draw definitive conclusions for all systems.
  • Future Work: The authors suggest that interactive systems that support domain experts in manual correction of terminological choices would be a promising direction. They also highlight the need for MT systems to be trained in ways that can directly address subtle domain-specific issues. The release of the test suite is intended to facilitate future research in this area.

Summary updated

BibTeX
@inproceedings{vojtechova2019sao,
  author = {Tereza Vojtěchová and Michal Novák and Miloš Klouček and Ondřej Bojar},
  title = {{SAO WMT19 Test Suite: Machine Translation of Audit Reports}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure