Efficient Methods for Mapping Neural Machine Translator on FPGAs

Qin Li, Xiaofan Zhang, Jinjun Xiong, Wen-Mei Hwu, Deming Chen

Abstract

Neural Machine Translation (NMT) has set new standards in language processing but at a significant computational cost. While GPUs are the conventional choice for deployment, their high power consumption limits edge applications. This paper details the first successful mapping of a large-scale, real-life NMT model onto an FPGA. Through a combination of mixed-precision quantization, a novel heterogeneous decoder architecture, and targeted optimization of the attention mechanism, the proposed system achieves a 13.1x speedup over standard FPGA implementations and 5x greater energy efficiency than modern GPUs, all while maintaining translation quality.

Introduction

Modern NMT models utilize complex structures like bidirectional Gated Recurrent Units (GRUs) and attention mechanisms, involving hundreds of millions of parameters. Mapping these 172 GFLOP models to FPGAs is challenging due to programming complexity and hardware constraints. Previous attempts were often limited to small-scale "toy" models. This research addresses these hurdles by providing a high-performance, energy-efficient hardware implementation that preserves BLEU scores.

Methodology and Hardware-Software Co-Design

The researchers utilized several strategies to bridge the gap between high-level NMT algorithms and low-level FPGA hardware:

  1. High-Level Synthesis (HLS): C/C++ based hardware design allowed for rapid iteration.
  2. Mixed-Precision Quantization: Weights and most calculations use 16-bit half-precision, while sensitive non-linear operations remain in 32-bit floating-point to ensure accuracy.
  3. Heterogeneous Decoder Design: The decoding process is split into specialized modules. Decoder One handles the initial word, while Decoder Five is optimized for the parallel paths required by beam search.
Figure 4: The heterogeneous decoder design separates the initial decoding step from subsequent beam search iterations to optimize resource usage.
  1. Refined Attention Module: By profiling the system, the authors identified the attention mechanism as the primary bottleneck. They optimized the calculation order to reuse intermediate results, significantly reducing redundant operations.

Key Insights: The Attention Bottleneck

Profiling revealed that the attention mechanism accounts for over 90% of the decoder's total computation. Focusing optimization efforts here—specifically through on-chip weight storage and result reuse—yielded the most significant performance gains.

Figure 5: Computational breakdown showing that the attention mechanism dominates the decoding process, representing over 90% of the workload.

Furthermore, the authors exploited the nature of beam search. While typically viewed as a computational burden, beam search increases the "compute-to-communication" (CTC) ratio. Since all candidates in a beam share the same weights, the hardware can load weights once and utilize them across all parallel paths, enhancing overall efficiency.

Experimental Results

The implementation was evaluated on the WMT14 English-to-French task using the Europarl dataset. Key performance metrics include:

  • Performance: 22.0 GFLOPS throughput and a 13.1x end-to-end speedup over baseline FPGA implementations.
  • Accuracy: A BLEU score of 22.6, slightly exceeding the PyTorch Float32 baseline (22.3).
  • Efficiency: 1.05 GFLOPS/Watt, representing a ~5x improvement in energy efficiency over comparable GPU and CPU setups.

Conclusion

This work demonstrates that FPGAs can effectively host large-scale NMT models with superior energy efficiency. While the industry is moving toward Transformer architectures, the fundamental optimizations developed here—such as MVM kernels and heterogeneous task partitioning—remain highly relevant. Future work will explore multi-head attention implementations and even lower bit-width quantization to further push the boundaries of hardware-accelerated translation.

Summary updated

Abstract

Neural machine translation (NMT) is one of the most critical applications in natural language processing (NLP) with the main idea of converting text in one language to another using deep neural networks. In recent year, we have seen continuous development of NMT by integrating more emerging technologies, such as bidirectional gated recurrent units (GRU), attention mechanisms, and beam-search algorithms, for improved translation quality. However, with the increasing problem size, the real-life NMT models have become much more complicated and difficult to implement on hardware for acceleration opportunities. In this article, we aim to exploit the capability of FPGAs to deliver highly efficient implementations for real-life NMT applications. We map the inference of a large-scale NMT model with total computation of 172 GFLOP to a highly optimized high-level synthesis (HLS) IP and integrate the IP into Xilinx VCU118 FPGA platform. The model has widely used key features for NMTs, including the bidirectional GRU layer, attention mechanism, and beam search. We quantize the model to mixed-precision representation in which parameters and portions of calculations are in 16-bit half precision, and others remain as 32-bit floating-point. Compared to the float NMT implementation on FPGA, we achieve 13.1× speedup with an end-to-end performance of 22.0 GFLOPS without any accuracy degradation. Based on our knowledge, this is the first work that successfully implements a real-life end-to-end NMT model to an FPGA on board.
BibTeX
@article{li2021efficient,
  author = {Qin Li and Xiaofan Zhang and Jinjun Xiong and Wen-Mei Hwu and Deming Chen},
  title = {{Efficient Methods for Mapping Neural Machine Translator on FPGAs}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2021},
  doi = {10.1109/tpds.2020.3047371},
}
Figure