Abstract: Alexandre Patry, Fabrizio Gotti, Philippe Langlais. Proceedings on the Workshop on Statistical Machine Translation.
2006.
Introduction
In the mid-2000s, phrase-based statistical machine translation (SMT) was the dominant paradigm. While many components of the SMT pipeline, such as word alignment (GIZA++) and language modeling (SRILM), were open-source, the "decoder"—the engine that searches for the best translation—remained a significant bottleneck.
The industry standard at the time, PHARAOH, was distributed only as a pre-compiled binary with a restrictive license. This "black box" nature prevented researchers from inspecting or modifying its inner workings, hindering scientific transparency. To address this, the authors developed RAMSES, an open-source, phrase-based decoder designed to be a functional clone of PHARAOH, built upon the MOOD (Modular Object-Oriented Decoder) framework.
Methods and Framework
The authors' contribution is twofold:
- MOOD Framework: A C++ framework designed to decouple search space exploration from model representation. This modularity allows researchers to swap out search strategies or scoring functions without rewriting the entire engine.
- RAMSES Implementation: A specific instance within MOOD designed to replicate PHARAOH’s behavior, supporting the same command-line syntax and output formats (including n-best lists).
To ensure a fair comparison, the authors used a standard log-linear model with eight coefficients, including translation table scores, language model scores, word penalties, and distortion models. These coefficients were tuned independently for both decoders using Minimum Error Rate Training (MERT).
Experimental Setup
The systems were evaluated using the WMT 2006 shared task datasets, covering language pairs between English and French, Spanish, and German. The evaluation set consisted of 2,000 sentence pairs for each translation direction.
Key Results
Quantitative Performance
RAMSES achieved translation quality nearly identical to PHARAOH across all tested language pairs. Using bootstrapping techniques to resample the test set 1,000 times, the authors demonstrated that the minor differences in BLEU scores were generally not statistically significant.
As shown in the performance data, RAMSES (R) often matched or slightly exceeded PHARAOH (P) in metrics like BLEU and precision ($p_1$ through $p_4$). For instance, in the English $\to$ French task, RAMSES achieved a BLEU score of 31.79 compared to PHARAOH's 30.96.
Human Evaluation
A blind study involving 100 translations per language pair was conducted to assess qualitative differences. Evaluators were asked to choose a preferred translation or indicate no preference.
The results indicated that in 48% to 71% of cases, human evaluators had no preference between the two decoders. Interestingly, when a preference was expressed for the Spanish $\to$ English and French $\to$ English tasks, RAMSES was frequently favored over the original PHARAOH binary.
Discussion and Limitations
While RAMSES successfully cloned PHARAOH's quality, it faced a significant trade-off in efficiency: the modular RAMSES was approximately twice as slow as the highly optimized, proprietary PHARAOH binary.
Furthermore, the authors noted that German remains a difficult target due to its complex morphology and word order, suggesting that future iterations of the MOOD framework should integrate better pre-processing techniques, such as compound splitting. Ultimately, the goal of MOOD is to provide a transparent platform for the community to develop and share entirely new decoding strategies beyond mere cloning.
Summary updated