Marian: Fast Neural Machine Translation in C++

Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, Alexandra Birch

Abstract

We present Marian, an efficient and self-contained Neural Machine Translation framework with an integrated automatic differentiation engine based on dynamic computation graphs. Marian is written entirely in C++. We describe the design of the encoder-decoder framework and demonstrate that a research-friendly toolkit can achieve high training and translation speed.

Introduction

Neural Machine Translation (NMT) has revolutionized the field of language translation but at a significant computational cost. Training state-of-the-art models often requires massive datasets and extensive GPU resources. While popular frameworks like TensorFlow and PyTorch provide the flexibility needed for research, they often introduce overhead due to their Python-centric design and general-purpose nature.

Marian addresses this by providing a "research-friendly" toolkit that does not compromise on raw performance. By implementing the entire framework in C++ and optimizing specifically for NMT tasks, the authors eliminate the performance bottlenecks typically associated with high-level language bindings.

System Architecture

Marian’s design is centered on four core technical pillars:

  1. Pure C++11 Implementation: By avoiding Python bindings, Marian allows for "barrier-free optimization." Researchers can write efficient C++ code for everything from model logic to custom GPU kernels, ensuring maximum performance across the entire stack.
  2. Custom Auto-Differentiation Engine: Marian features a built-in back-end for reverse-mode automatic differentiation using dynamic computation graphs. This approach is particularly well-suited for NMT, where input sequences vary in length, requiring a flexible graph structure for every batch.
  3. NMT-Specific Optimizations: The engine is specifically tuned for translation. It utilizes fused operators for RNN cells and attention mechanisms, alongside an atomic layer-normalization operator. These optimizations are designed to minimize memory bandwidth bottlenecks, a common hurdle in deep learning.
  4. Extensible API: Despite being low-level, the framework offers a simplified C++ API for building models. This modularity allows researchers to easily combine different components, such as a Transformer decoder with an RNN encoder.

Key Insights and Ideas

A major takeaway from Marian is the power of specialization. By focusing exclusively on NMT, the framework can implement "fused" kernels that combine multiple operations into a single GPU call, significantly reducing execution overhead.

Furthermore, Marian demonstrates that dynamic graphs can be highly efficient. While static graphs were historically preferred for speed, Marian’s optimized dynamic implementation proves that flexibility does not have to come at the cost of performance. The framework also introduces memory-centric batching, which automatically adjusts batch sizes to fully utilize available GPU memory, removing the need for manual tuning by the user.

Key Results

Marian demonstrates exceptional performance gains over existing toolkits. On a single GPU, Marian achieved training speeds approximately 4x faster than Nematus. This performance gap widens significantly in multi-GPU environments; on an 8-GPU setup, Marian was 30x faster than Nematus for identical models, largely due to its highly efficient scaling.

Multi-GPU scaling of training throughput (source tokens per second) for various NMT architectures in Marian.

As shown in the figure above, Marian exhibits near-linear scaling across multiple GPUs for Shallow RNN, Deep RNN, and Transformer architectures. This efficiency allowed the team to replicate the winning WMT2017 English-German system and even improve upon it by 1.2 BLEU points by integrating a Transformer architecture. Additionally, Marian-based models set new benchmarks in Grammatical Error Correction (GEC), outperforming previous neural systems by over 8% $M^2$ score on the CoNLL-2014 benchmark.

Limitations and Future Work

While Marian excels on GPUs, the authors acknowledge that CPU-bound computation remains an area for further optimization. Additionally, the pure C++ nature of the toolkit presents a steeper learning curve for researchers accustomed to Python. Future development will focus on "auto-batching" to simplify model code and even more aggressive kernel fusion to maintain its performance lead.

Summary updated

Abstract

We present Marian, an efficient and self-contained Neural Machine Translation framework with an integrated automatic differentiation engine based on dynamic computation graphs. Marian is written entirely in C++. We describe the design of the encoder-decoder framework and demonstrate that a research-friendly toolkit can achieve high training and translation speed.
BibTeX
@inproceedings{junczysdowmunt2018marian,
  author = {Marcin Junczys-Dowmunt and Roman Grundkiewicz and Tomasz Dwojak and Hieu Hoang and Kenneth Heafield and Tom Neckermann and Frank Seide and Ulrich Germann and Alham Fikri Aji and Nikolay Bogoychev and André F. T. Martins and Alexandra Birch},
  title = {{Marian: Fast Neural Machine Translation in C++}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=ihK49_SavI7},
}
Figure