Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU

Jacob DevlinEMNLP 2017 · DOI

Abstract

Neural Machine Translation (NMT) has significantly improved translation quality but at a high computational cost, typically necessitating GPUs for real-time performance. This paper presents a series of engineering and architectural optimizations that enable high-quality NMT decoding on standard single-threaded CPUs. By utilizing 16-bit integer quantization and a hybrid RNN-Fully Connected architecture, the authors achieve speeds of over 100 words per second and a BLEU score of 38.3 on the WMT English-French task, matching state-of-the-art performance with significantly lower latency.

Introduction

While NMT has revolutionized the field, its deployment in production environments is often hindered by the "computational tax" of deep attentional models. Standard attentional sequence-to-sequence models are often an order of magnitude slower than the phrase-based systems they replaced. Standard CPUs often struggle to decode more than 10 words per second, forcing a choice between expensive specialized hardware or slower translation. This work, authored by Jacob Devlin, aims to bridge this gap, enabling "sharp" (accurate) models to run efficiently on "dull" (standard CPU) hardware.

Engineering Optimizations

The authors first address the efficiency of the decoder itself through several C++ optimizations that preserve mathematical output while reducing latency.

16-Bit Integer Quantization

A primary bottleneck in NMT is matrix multiplication. By switching from 32-bit floating-point to 16-bit fixed-point integer math, the system leverages CPU SIMD (Single Instruction, Multiple Data) instructions. This provides a 2x-3x speedup over optimized libraries like Intel MKL with virtually zero impact on BLEU scores (less than 0.02 difference), suggesting that NMT models are remarkably robust to quantization.

Pre-computation and State Merging

Further gains are achieved by pre-calculating static components like word embeddings and attention context that do not change during the decoding of a specific sentence. Additionally, the decoder identifies identical recurrent states across different hypotheses in a beam search, performing expensive matrix multiplications only once per unique state.

Table 1: Cumulative speedup from sequential engineering optimizations on a single-threaded CPU.

Architectural Innovation: The Hybrid Model

Deep Recurrent Neural Networks (RNNs) are computationally expensive because each layer is dependent on the previous timestep's output. The paper proposes a Mixed RNN+FC architecture to mitigate this:

  • Contextual Layer: A single RNN (GRU or LSTM) layer handles sentence context and attention.
  • Deep Stack: Subsequent layers are replaced with stacked Fully-Connected (FC) layers using ResNet-style skip connections.

Since an FC layer is roughly one-sixth the computational cost of a GRU layer, this hybrid approach maintains the representational power of deep models at a fraction of the computational cost. This design reflects the insight that recurrence is only strictly necessary for tracking state and attention; subsequent depth can be handled by cheaper feed-forward layers.

Key Results

The proposed methods were evaluated on the WMT English-French NewsTest2014 benchmark. The optimized system achieved a throughput of 100 words/second on a single-threaded CPU, a significant improvement over previous state-of-the-art systems that often required 44 CPU cores to reach similar speeds.

Table 2: Comparison of translation quality (BLEU) and decoding speed (Words/Sec) against various state-of-the-art NMT systems.

As shown in the results, the best-performing system—an ensemble of two hybrid models (E1)—reached 38.3 BLEU. This performance is highly competitive with benchmarks like Google NMT while remaining significantly faster to decode on standard hardware. The cumulative effect of these optimizations highlights how much performance is often left on the table by standard deep learning frameworks when deploying to CPUs.

Conclusion

This work demonstrates that high-quality translation can be made accessible on standard CPU hardware. By optimizing both the software implementation (through 16-bit math and state merging) and the model structure (through the hybrid RNN+FC architecture), the authors provide a viable path for large-scale NMT deployment without the need for specialized AI accelerators.

Summary updated

BibTeX
@inproceedings{devlin2017sharp,
  author = {Jacob Devlin},
  title = {{Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1300},
}
Figure