MT-ComparEval: Graphical evaluation interface for Machine Translation development

Ondrej Klejch, Eleftherios Avramidis, Aljoscha Burchardt, Martin PopelPrague Bull. Math. Linguistics 2015 · OpenReview

Introduction

Evaluating Machine Translation (MT) systems is a complex task that extends far beyond simple aggregate scores. While metrics like BLEU provide a high-level performance indicator, they often function as "black boxes," failing to explain why one model outperforms another or where specific regressions occur. In a typical development cycle, researchers may run dozens of experiments, making the manual tracking of linguistic patterns—such as improvements in local idioms versus failures in long-distance dependencies—an error-prone and labor-intensive process.

To address these challenges, the authors developed MT-ComparEval, an open-source, web-based tool designed to provide a comprehensive graphical interface for MT evaluation. It allows developers to move beyond summary statistics by visualizing automatic metrics, performing statistical significance tests, and drilling down into sentence-level differences.

System Architecture

MT-ComparEval is built on the PHP Nette framework and utilizes SQLite for data management. Its architecture is designed for ease of use and integration into existing workflows through three primary components:

  1. The Evaluation Panel (GUI): A web-based interface where users can browse experiments, compare system outputs side-by-side, and analyze statistical charts.
  2. The Back-end "Watcher": A background process that monitors designated directories for new translation outputs. Upon detection, it automatically imports the data and triggers the evaluation pipeline.
  3. The Evaluation Mechanism: This engine calculates standard metrics, including BLEU, Precision, Recall, and F-score. Crucially, it implements bootstrap resampling—generating 1,000 resamples of the test set to calculate confidence intervals and p-values, ensuring that observed gains are statistically significant.
The MT-ComparEval dashboard showing metric scores across multiple translation tasks.

Visual Analysis and Error Detection

One of the tool's most powerful features is its ability to facilitate deep error analysis through visual comparison. Unlike previous tools tied to specific pipelines, MT-ComparEval is pipeline-agnostic, allowing for the comparison of rule-based, statistical (SMT), and neural (NMT) models in a single view.

Sentence-Level Comparisons

The interface employs sophisticated color-coding to highlight differences between system outputs and human references. "Confirmed n-grams" (matches with the reference) and "improving/worsening n-grams" are visually distinguished, allowing researchers to see exactly how a model's output has changed relative to a baseline.

Sentence-level comparison view with visual highlighting of n-gram differences and local metrics.

Key Analytical Features

  • Sorting by Delta: Users can sort sentences by the difference in their scores. This highlights "regressions"—cases where a generally superior model performs poorly on specific segments—enabling targeted debugging.
  • N-gram Winning/Losing Panes: The tool identifies the top 10 n-grams that contribute most to a system's relative performance, revealing whether a model consistently struggles with specific phrases or functional words.

Experimental Validation

The tool was demonstrated using system outputs from the WMT (Workshop on Machine Translation) Shared Tasks 2014 and 2015. In the "Newstest 2015 en-cs" (English to Czech) task, the tool successfully integrated paired bootstrap resampling. This provided a one-tailed p-value directly in the interface, helping researchers determine if an apparent improvement (e.g., a small BLEU gain) was statistically robust or merely noise.

Conclusion and Future Work

MT-ComparEval significantly reduces the bookkeeping overhead for MT researchers by automating metric calculation and comparison visualization. While the current version focuses on the BLEU/F-score family, future development aims to include a broader range of metrics (such as METEOR or TER) and deeper linguistic analysis, such as part-of-speech based error tracking. Additionally, plans for multi-user support and enhanced export functions (CSV, LaTeX) will further improve its utility for collaborative research environments.

Summary updated

BibTeX
@inproceedings{klejch2015mt,
  author = {Ondrej Klejch and Eleftherios Avramidis and Aljoscha Burchardt and Martin Popel},
  title = {{MT-ComparEval: Graphical evaluation interface for Machine Translation development}},
  booktitle = {Prague Bull. Math. Linguistics},
  year = {2015},
  url = {https://openreview.net/forum?id=EXjvGBBmY3J},
}
Figure