Abstract
Neural Machine Translation (NMT) has set new standards in language processing but at a significant computational cost. While GPUs are the conventional choice for deployment, their high power consumption limits edge applications. This paper details the first successful mapping of a large-scale, real-life NMT model onto an FPGA. Through a combination of mixed-precision quantization, a novel heterogeneous decoder architecture, and targeted optimization of the attention mechanism, the proposed system achieves a 13.1x speedup over standard FPGA implementations and 5x greater energy efficiency than modern GPUs, all while maintaining translation quality.
Introduction
Modern NMT models utilize complex structures like bidirectional Gated Recurrent Units (GRUs) and attention mechanisms, involving hundreds of millions of parameters. Mapping these 172 GFLOP models to FPGAs is challenging due to programming complexity and hardware constraints. Previous attempts were often limited to small-scale "toy" models. This research addresses these hurdles by providing a high-performance, energy-efficient hardware implementation that preserves BLEU scores.
Methodology and Hardware-Software Co-Design
The researchers utilized several strategies to bridge the gap between high-level NMT algorithms and low-level FPGA hardware:
- High-Level Synthesis (HLS): C/C++ based hardware design allowed for rapid iteration.
- Mixed-Precision Quantization: Weights and most calculations use 16-bit half-precision, while sensitive non-linear operations remain in 32-bit floating-point to ensure accuracy.
- Heterogeneous Decoder Design: The decoding process is split into specialized modules. Decoder One handles the initial word, while Decoder Five is optimized for the parallel paths required by beam search.
- Refined Attention Module: By profiling the system, the authors identified the attention mechanism as the primary bottleneck. They optimized the calculation order to reuse intermediate results, significantly reducing redundant operations.
Key Insights: The Attention Bottleneck
Profiling revealed that the attention mechanism accounts for over 90% of the decoder's total computation. Focusing optimization efforts here—specifically through on-chip weight storage and result reuse—yielded the most significant performance gains.
Furthermore, the authors exploited the nature of beam search. While typically viewed as a computational burden, beam search increases the "compute-to-communication" (CTC) ratio. Since all candidates in a beam share the same weights, the hardware can load weights once and utilize them across all parallel paths, enhancing overall efficiency.
Experimental Results
The implementation was evaluated on the WMT14 English-to-French task using the Europarl dataset. Key performance metrics include:
- Performance: 22.0 GFLOPS throughput and a 13.1x end-to-end speedup over baseline FPGA implementations.
- Accuracy: A BLEU score of 22.6, slightly exceeding the PyTorch Float32 baseline (22.3).
- Efficiency: 1.05 GFLOPS/Watt, representing a ~5x improvement in energy efficiency over comparable GPU and CPU setups.
Conclusion
This work demonstrates that FPGAs can effectively host large-scale NMT models with superior energy efficiency. While the industry is moving toward Transformer architectures, the fundamental optimizations developed here—such as MVM kernels and heterogeneous task partitioning—remain highly relevant. Future work will explore multi-head attention implementations and even lower bit-width quantization to further push the boundaries of hardware-accelerated translation.
Summary updated