Adding Multi-Threaded Decoding to Moses

Barry HaddowPrague Bull. Math. Linguistics 2010 · OpenReview

Introduction

In the late 2000s, the "multi-core revolution" transformed computing, yet the Moses decoder—the industry standard for open-source machine translation (MT)—remained primarily single-threaded. To leverage multiple cores, researchers typically relied on external scripts like moses-parallel.pl, which split input files and launched multiple independent processes. This approach, however, was plagued by massive memory overhead. Statistical MT models are large; loading separate copies into RAM for every process quickly exhausts system resources on high-core-count servers. This paper details the architectural overhaul of Moses to support native multi-threaded decoding, enabling a shared-memory "thread-per-sentence" approach.

Implementation Strategy

The primary goal was to transition Moses from a memory-heavy "process-per-segment" model to a more efficient multi-threaded architecture using the C++ Boost libraries. This required a significant refactoring of a complex legacy codebase to ensure thread safety.

Refactoring Global State

Moses originally utilized a global singleton, StaticData, to store configuration and model information. To prevent threads from overwriting each other's metadata, sentence-specific data was moved into a new Manager object instantiated for each thread.

Concurrency and Synchronization

To manage shared data structures, such as vocabulary caches, the author implemented reader-writer locks. This allows multiple threads to read data simultaneously while ensuring exclusive access for write operations. For binarized on-disk models, thread-local storage was employed to provide each thread with a private buffer, minimizing locking overhead.

ThreadPool Architecture

A central ThreadPool was introduced to manage a queue of translation tasks. Because sentences might finish out of order (e.g., a short sentence #10 finishing before a complex sentence #1), a new OutputCollector class was implemented to reorder finished translations, ensuring the output matches the input sequence.

Sequence diagram of the multi-threaded decoding architecture.

Evaluation and Performance

The implementation was validated using the WMT2009 French-English news test set (3,027 sentences) and the Europarl French-English corpus for Minimum Error Rate Training (MERT).

Decoding Speedup

On a 16-core server, the multi-threaded implementation achieved a 3.5x speedup using 6 threads compared to the single-threaded baseline. As shown in the performance data, while the speedup is significant, it is sub-linear. This indicates resource contention, where threads occasionally wait on shared locks or compete for hardware bandwidth (RAM/disk).

Decoding performance across different thread counts.

Impact on Research Workflow

For MT researchers, the efficiency gains in MERT are particularly impactful. Using 4 threads reduced the time required for MERT iterations by approximately 50%, significantly accelerating the model development cycle.

Key Insights and Future Work

The transition to multi-threading was driven as much by memory constraints as by CPU utilization. By sharing heavy model data across threads, the system can support more concurrent workers within the same RAM footprint. This architectural shift also enabled the creation of the "Moses Server," an XML-RPC wrapper that allows Moses to function as a live service for concurrent web requests.

Despite these successes, limitations remain. Scaling beyond 6 threads showed diminishing returns, suggesting a need for deeper investigation into lock contention. Additionally, certain components like "generation tables" were not yet thread-safe at the time of publication. Future work includes enabling the Moses server to host multiple language pairs simultaneously through configuration switching, further optimizing resource usage.

Summary updated

BibTeX
@inproceedings{haddow2010adding,
  author = {Barry Haddow},
  title = {{Adding Multi-Threaded Decoding to Moses}},
  booktitle = {Prague Bull. Math. Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=4Mu8G8mpzwg},
}
Figure