Improved Minimum Error Rate Training in Moses

Nicola Bertoldi, Barry Haddow, Jean-Baptiste FouetPrague Bull. Math. Linguistics 2009 · OpenReview

Introduction

Minimum Error Rate Training (MERT) is a cornerstone of Statistical Machine Translation (SMT), used to tune the weights of various models during decoding. Historically, the implementation within the Moses toolkit was a complex assembly of scripts in multiple languages, making it difficult to maintain or extend. This paper presents a complete rewrite of the MERT module, transitioning it to a modular C++ framework designed for flexibility and research experimentation.

System Architecture

The new implementation moves away from the "Frankenstein's monster" of legacy scripts toward a clean, object-oriented design. The architecture is centered around two primary abstract classes: the Scorer and the Optimizer.

The MERT workflow, illustrating the interaction between the decoder (Moses), the Scorer, and the Optimizer within inner and outer loops.

As shown in the workflow above, the process is bifurcated into an inner loop for optimization and an outer loop that involves the decoder. To support this, the authors introduced two standalone tools:

  1. extractor: This tool processes n-best translation lists to generate scoring statistics and feature values, saving them in a compact binary format.
  2. mert: This tool performs the actual weight optimization using the pre-extracted data.
UML class diagram of the modular MERT architecture, highlighting the Scorer and Optimizer hierarchies.

The modularity shown in the class diagram allows researchers to easily swap components. For instance, a new translation metric can be implemented by extending the Scorer class, while a new search strategy can be added via the Optimizer class. This separation allows the "inner loop" of optimization to run much faster without constantly re-reading raw text files.

Experimental Setup and Results

The framework was evaluated using datasets from the WMT08 workshop, including French-English, German-English, and Spanish-English pairs.

Performance and Convergence

The new implementation achieved BLEU scores nearly identical to the legacy system (e.g., 25.70 vs 25.55 on news-test07), validating its mathematical correctness. Convergence is typically reached within 5–6 iterations when utilizing all previous n-best lists.

Extensibility: Regularization

To demonstrate the framework's flexibility, the authors implemented a "smoothed" BLEU metric (regularization). While this did not yield significant gains for the tested European language pairs, the ease of implementation—requiring only a straightforward addition to the StatisticsBasedScorer class—confirmed the benefits of the new architecture.

Discussion and Future Work

While the new system offers superior modularity, it initially faced efficiency challenges. Specifically, the lack of duplicate removal for translation candidates led to increased disk usage and slower execution compared to the highly optimized legacy scripts.

Future developments for the Moses MERT module include:

  • Efficiency Improvements: Implementing duplicate filtering and parallelization to reduce the computational footprint.
  • Metric Support: Expanding the library of scorers to include WER and NIST.
  • Lattice Optimization: Extending the tuner to optimize over translation lattices rather than just n-best lists, providing a richer search space for weight tuning.

This rewrite provides the SMT community with a robust, extensible sandbox, paving the way for more sophisticated optimization techniques in machine translation.

Summary updated

BibTeX
@inproceedings{bertoldi2009improved,
  author = {Nicola Bertoldi and Barry Haddow and Jean-Baptiste Fouet},
  title = {{Improved Minimum Error Rate Training in Moses}},
  booktitle = {Prague Bull. Math. Linguistics},
  year = {2009},
  url = {https://openreview.net/forum?id=b4jmcjLFUS},
}
Figure