SGNMT - A Flexible NMT Decoding Platform for Quick Prototyping of New Models and Search Strategies

Felix Stahlberg, Eva Hasler, Danielle Saunders, Bill Byrne

Introduction

Neural Machine Translation (NMT) has seen rapid advancements, yet experimenting with complex architectures—such as ensembling disparate models or enforcing symbolic constraints—often requires significant engineering effort. Standard decoding frameworks are frequently monolithic, tightly coupling the search algorithm with the underlying model. SGNMT (Syntactically Guided NMT) addresses this by providing an open-source platform that decouples translation models ("predictors") from search algorithms ("decoders"). This modularity enables researchers to rapidly prototype systems by mixing neural models with symbolic constraints like lattices or grammars.

The core innovation of SGNMT is its dual-concept architecture, which separates the "what" (scoring) from the "how" (searching). This separation is achieved through a standardized interface that allows any search strategy to interact with any scoring module.

The modular architecture of SGNMT decouples decoders from predictors through a standardized interface.

Predictors (The "What")

Predictors are scoring modules that provide a probability distribution over the target vocabulary at each step. SGNMT defines a generic interface with methods such as initialize, predict_next, and consume. This abstraction allows the platform to wrap diverse models, from neural frameworks like TensorFlow and Theano/Blocks to symbolic models like Finite State Transducers (FSTs) and n-gram language models.

SGNMT supports a wide variety of predictors, ranging from neural models to symbolic constraints.

Decoders (The "How")

Decoders are the search algorithms that traverse the hypothesis space defined by the predictors. Because the interface is standardized, any decoder can work with any combination of predictors. SGNMT includes standard strategies like Beam Search and Greedy decoding, alongside specialized algorithms:

  • A* Search: Uses heuristic functions defined via predictors to guide the search.
  • Depth-First Search (DFS): Efficiently enumerates complete search spaces, which is particularly useful for exhaustive FST-based rescoring.
  • Syncbeam: A specialized beam search that synchronizes models with different tokenization schemes (e.g., word-based vs. character-based) by comparing scores only when they reach a synchronization boundary.

Performance and Scalability

While SGNMT prioritizes flexibility, it remains competitive in terms of performance. To mitigate the overhead of its Python-based architecture, the platform introduces Batch Decoding. A multi-threaded scheduler groups sentences into large GPU jobs, significantly boosting throughput. On the WMT English-French task using a Titan X GPU, SGNMT achieved a decoding speed of 911.6 words per second. While specialized C++ engines like Marian-NMT are faster, SGNMT's speed is more than sufficient for research and prototyping.

Key Insights and Applications

SGNMT's strength lies in its ability to provide "Syntactic Guidance." By using symbolic constraints (like FSTs) to guide neural models, researchers can perform tasks like lattice rescoring or restricting output to specific vocabularies.

A notable feature is the handling of tokenization mismatches via the syncbeam decoder. This allows for ensembling models that operate on different subword units. Furthermore, the platform's flexibility has enabled creative applications, such as automatic music composition using LSTMs for creativity and Weighted Finite State Automata (WFSAs) to enforce musical constraints in the style of Bach chorales.

In practical evaluations, the platform demonstrated its utility in system-level combinations. By using the sepbeam strategy to combine three BPE-based NMT models, researchers reported a gain of 0.5 BLEU over a single system on the ASPEC Japanese-English task.

Conclusion

SGNMT serves as a bridge between high-performance production engines and the need for rapid research prototyping. By providing a framework-agnostic platform where scoring and search are strictly separated, it minimizes the implementation effort required for new ideas. It has proven particularly effective in academic settings, allowing students and researchers to implement complex projects without building decoders from scratch.

Summary updated

Abstract

This paper introduces SGNMT, our experimental platform for machine translation research. SGNMT provides a generic interface to neural and symbolic scoring modules (predictors) with left-to-right semantic such as translation models like NMT, language models, translation lattices, $n$-best lists or other kinds of scores and constraints. Predictors can be combined with other predictors to form complex decoding tasks. SGNMT implements a number of search strategies for traversing the space spanned by the predictors which are appropriate for different predictor constellations. Adding new predictors or decoding strategies is particularly easy, making it a very efficient tool for prototyping new research ideas. SGNMT is actively being used by students in the MPhil program in Machine Learning, Speech and Language Technology at the University of Cambridge for course work and theses, as well as for most of the research work in our group.
BibTeX
@inproceedings{stahlberg2017sgnmt,
  author = {Felix Stahlberg and Eva Hasler and Danielle Saunders and Bill Byrne},
  title = {{SGNMT - A Flexible NMT Decoding Platform for Quick Prototyping of New Models and Search Strategies}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-2005},
}
Figure