Design of the Moses Decoder for Statistical Machine Translation

Hieu Hoang, Philipp KoehnSETQALNLP 2008 · OpenReview

Abstract

We present a description of the implementation of the open source decoder for statistical machine translation which has become popular with many researchers in SMT research. The goal of the project is to create an open, high quality phrase-based decoder which can reduce the time and barrier to entry for researchers wishing to do SMT research. We discuss the major design objective for the Moses decoder, its performance relative to other SMT decoders, and the steps we are taking to ensure that its success will continue.

Introduction

In the mid-2000s, Statistical Machine Translation (SMT) faced a significant bottleneck. While theoretical advancements were rapid, software implementations were often proprietary or distributed as closed binaries. This "black box" nature prevented researchers from inspecting, modifying, or extending decoders with new features like Part-of-Speech (POS) tags. Moses was developed to address this "barrier to entry," providing an open, extensible, and well-documented platform intended to become the industry standard for SMT research.

Methods

Moses is implemented as a modular C++ library, prioritizing both performance and flexibility. Its design incorporates several key technical innovations:

Factored Translation Models

Unlike traditional models that treat words as atomic strings, Moses supports "factored" models. Words are represented as vectors of factors, such as the surface form, lemma, and POS tag. This allows the decoder to integrate linguistic information directly into the translation process, improving grammatical coherence.

Abstracted and Pluggable Components

The architecture is centered around a modular design using a ScoreProducer base class. New features, such as Language Models (LM) or Lexical Reordering models, can be integrated by inheriting from this class. This "pluggable" system allows researchers to easily experiment with new scoring components.

The ScoreProducer class hierarchy enables a modular, pluggable architecture for various scoring models.

Diverse Input Support

To accommodate various upstream sources, such as Automatic Speech Recognition (ASR), Moses supports multiple input types beyond simple text sentences. It can decode Confusion Networks and Word Lattices, which represent multiple weighted hypotheses.

InputType hierarchy supporting sentences, confusion networks, and word lattices.

Memory Management

Moses provides two phrase-table implementations: an in-memory version for maximum speed and a binary, on-disk version that loads data on-demand, enabling the use of massive datasets on machines with limited RAM.

Evaluation and Datasets

The toolkit was evaluated on a French-to-English translation task using 2,000 sentences from the News-Commentary corpus (WMT 2007). Moses demonstrated its community impact early on; by IWSLT 2007, approximately half of the participating teams had adopted the toolkit.

Key Results

Empirical tests showed that Moses significantly outperformed existing tools:

  • Speed: In a 2,000-sentence benchmark, Moses completed decoding in 69 minutes, compared to 99 minutes for Pharaoh, while maintaining an identical BLEU score of 19.57.
  • Efficiency: Compared to the Java-based Phramer, Moses was nearly 10x faster and used roughly 8x less memory (154MB vs. 1218MB).
  • Extensibility: The successful integration of four different language model implementations by the community shortly after release validated the modular design.

Discussion and Key Insights

The success of Moses stems from its "One-Stop Shop" strategy. It provides a complete ecosystem, including scripts for word alignment (GIZA++), phrase table generation, and parameter tuning (MERT). Furthermore, the authors emphasized platform agnosticism. Despite the field's Linux-centric nature, supporting Windows and Mac was crucial; early data showed 58% of website visitors were on Windows, indicating that cross-platform support effectively lowered the barrier for a broader range of users.

Limitations and Future Work

Current limitations include the need for structural changes to support emerging paradigms like hierarchical phrase-based models and discriminative models with very large feature sets. Additionally, the peripheral "glue" scripts require re-engineering to match the modularity of the C++ core. Future directions include applying the Moses engine to tasks like OCR correction and transliteration.

Summary updated

Abstract

We present a description of the implementation of the open source decoder for statistical machine translation which has become popular with many researchers in SMT research. The goal of the project is to create an open, high quality phrase-based decoder which can reduce the time and barrier to entry for researchers wishing to do SMT research. We discuss the major design objective for the Moses decoder, its performance relative to other SMT decoders, and the steps we are taking to ensure that its success will continue.
BibTeX
@inproceedings{hoang2008design,
  author = {Hieu Hoang and Philipp Koehn},
  title = {{Design of the Moses Decoder for Statistical Machine Translation}},
  booktitle = {SETQALNLP},
  year = {2008},
  url = {https://openreview.net/forum?id=BkZrBhxObr},
}
Figure