A Web Application for the Diagnostic Evaluation of Machine Translation over Specific Linguistic Phenomena

Antonio Toral, Sudip Kumar Naskar, Joris Vreeke, Federico Gaspari, Declan GrovesNAACL 2013

TL;DR

This paper introduces DELIC4MT, a web application and service that moves beyond generic machine translation (MT) evaluation metrics like BLEU. It allows users to pinpoint specific linguistic weaknesses in MT systems by evaluating performance on user-defined linguistic phenomena, offering much-needed diagnostic insights for improvement.

What Problem Does This Paper Address?

Traditional automatic MT evaluation metrics, such as BLEU and METEOR, provide a single, overall score for translation quality. While cheap and fast, these scores are often crude and lack diagnostic value. They tell you if a translation is bad, but not why. This makes it difficult for MT developers to understand specific failure modes (e.g., problems with word choice, sentence structure, or meaning) and for users to select the best MT system for a particular task. Human evaluation is more reliable and diagnostic but is expensive and time-consuming. The authors address this gap by proposing a method to evaluate MT performance on specific linguistic phenomena, providing fine-grained, linguistically motivated feedback.

Methods

The core of this work is the DELIC4MT (Diagnostic Evaluation of Linguistic Checkpoints for Machine Translation) framework. DELIC4MT allows for the evaluation of MT systems based on user-defined "linguistic checkpoints." These checkpoints are essentially patterns or features that appear in the source language and are expected to be translated correctly. They can be defined at various linguistic levels:

  • Lexical: Words with multiple possible translations (e.g., "de" in Spanish can be "of" or "from" in English).
  • Syntactic: Difficult sentence structures that require reordering or specific grammatical transformations (e.g., Spanish noun-preposition-noun structures).
  • Semantic: Words with multiple meanings (polysemous words).

DELIC4MT uses a procedure that involves:

  1. Identifying occurrences of the linguistic checkpoint in the source text.
  2. Using word alignment information to find the corresponding tokens in the target text.
  3. Comparing the MT system's output for these target tokens against a reference translation to see if the correct translation of the checkpoint is present.

The authors have implemented this framework as both a SOAP-compliant web service and a web application. The web service takes inputs like word alignments, the linguistic checkpoint definition (in Kybot format), the MT output, and the source/reference texts. The web application provides a user-friendly graphical interface to visualize the results, showing performance for each individual checkpoint instance.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation. However, it mentions that DELIC4MT has been tested on language pairs involving Arabic, Bulgarian, Dutch, English, French, German, Hindi, Italian, Turkish, and Welsh. The evaluation process relies on:

  • Source and target sides of a testset: These are required in KAF (Knowledge Annotation Format) format.
  • Gold standard (reference) translations: Used to compare against the MT output.
  • Word alignment information: Typically generated by tools like GIZA++.

Key Insights and Ideas

  • Beyond Overall Scores: The most significant insight is the move away from single, aggregated MT evaluation scores towards a diagnostic approach. This allows for a deeper understanding of where and why MT systems fail.
  • Linguistic Checkpoints as a Flexible Tool: The concept of linguistic checkpoints is powerful because it's highly flexible. Users can define checkpoints tailored to their specific needs, whether it's a common lexical ambiguity, a tricky syntactic construction, or a semantically ambiguous word.
  • Web Services for Accessibility: Packaging DELIC4MT as a web service and application significantly lowers the barrier to entry. Users, including non-technical ones like translators, don't need to install, configure, or maintain complex software, making advanced diagnostic evaluation accessible.
  • Fine-Grained Feedback: The ability to see performance on each occurrence of a checkpoint is crucial. This allows for detailed analysis and targeted improvements.

Key Results

The paper demonstrates the functionality of the DELIC4MT web application and service. While specific quantitative results on MT system performance are not the primary focus of this demonstration paper, the figures and descriptions illustrate:

  • Visualisation of Results: Figure 2 shows a screenshot of the web application displaying results for a "noun" checkpoint. It clearly indicates which checkpoint instances were matched (1/1) and which were missed (0/1) in the MT output, along with the overall recall score.
  • Diagnostic Information: The example shows how the system can identify specific translation errors, such as the failure to translate "monsieur" correctly in one instance, while succeeding in another.
  • Overall Recall: The example also shows an overall recall score (e.g., 0.5025), which is calculated by summing up successful checkpoint matches across all instances.

Limitations and Future Work

  • Language Pair Dependency: While DELIC4MT is designed to be portable to any language pair, the creation of effective linguistic checkpoints and the availability of necessary resources (like word aligners and KAF-formatted data) can vary significantly between languages.
  • Checkpoint Definition Complexity: Defining effective and comprehensive linguistic checkpoints can still require linguistic expertise.
  • Future Work: The authors suggest that future work could involve developing more sophisticated methods for automatically generating linguistic checkpoints and exploring their use in active learning scenarios for MT system development. They also highlight the potential for integrating DELIC4MT with other MT evaluation tools and platforms.

Summary updated

BibTeX
@inproceedings{toral2013web,
  author = {Antonio Toral and Sudip Kumar Naskar and Joris Vreeke and Federico Gaspari and Declan Groves},
  title = {{A Web Application for the Diagnostic Evaluation of Machine Translation over Specific Linguistic Phenomena}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure