Abstract
Deploying Neural Machine Translation (NMT) models on mobile devices is critical for ensuring user privacy, achieving low latency, and enabling offline functionality. However, the high capacity of modern NMT models typically results in large footprints that exceed the storage, memory, and power constraints of mobile hardware. Existing compression techniques often prioritize theoretical metrics like FLOPs or general-purpose inference engines that struggle with the auto-regressive nature of decoding. In this paper, we present MobileNMT, a co-designed system that achieves high-quality translation within a 15MB memory footprint and 30ms latency. By combining architectural principles optimized for quantization with a custom INT8 inference engine, MobileNMT delivers a 47.0x speedup and 99.5% memory savings compared to standard systems, while retaining over 88% of the translation quality of a "Transformer-Big" model.
Introduction
Neural Machine Translation has reached impressive accuracy levels by scaling models to hundreds of millions of parameters. While effective on servers, these "Big" models are impractical for mobile deployment. They consume excessive storage, drain batteries through high computational demands, and suffer from high latency during the step-by-step (auto-regressive) decoding process required for text generation.
Bridging the gap between theoretical compression and industrial-grade deployment requires more than just reducing FLOPs. It requires addressing memory-mapped I/O (MMI/O) and hardware-specific instruction sets. MobileNMT addresses these challenges through a three-pronged approach: hardware-aware architecture design, quantization-friendly training, and a custom-built inference engine.
System Design
1. Architectural Principles
The researchers identified three core principles to balance model size and translation quality:
- Vocabulary Scaling: Rather than using complex embedding factorization, the authors found that simply reducing the vocabulary size via Byte-Pair Encoding (BPE) is the most effective way to shrink the embedding layer for small-scale models.
- Width vs. Sharing: While parameter sharing is a common compression tactic, it often results in wider weight distributions that are difficult to quantize. MobileNMT instead reduces the model width (hidden dimension), which keeps numerical ranges tight and "quantization-friendly."
- Asymmetric Depth: In mobile NMT, the decoder is the primary bottleneck because it must run once for every generated word. MobileNMT utilizes a deep encoder (12 layers) to capture complex semantics and a very shallow decoder (2 layers) to minimize the recurring cost of auto-regressive generation.
2. Training Strategies
To recover the accuracy lost by shrinking the model, MobileNMT employs several advanced training techniques:
- Sequence-level Knowledge Distillation: A large "Teacher" model (Transformer-Big) is used to train the small "Student" model, guiding it to mimic the output distributions of the larger network.
- Quantization-Aware Training (QAT) with L2 Regularization: The authors use L2 regularization specifically to "squash" weight ranges. This forces weights into a smaller numerical space, making them easier to represent as 8-bit integers (INT8) with minimal precision loss.
- Dropout Removal: For parameter-limited models, the researchers discovered that removing dropout actually improved performance by nearly 2 BLEU points, as every parameter is needed to represent the underlying data distribution.
3. The Lightseq-ARM Inference Engine
The system includes a custom engine optimized for ARM CPUs, the standard architecture for mobile devices. Key features include:
- SIMD Optimization: Utilizing Single Instruction, Multiple Data (SIMD) instructions to process multiple 8-bit integers in parallel.
- Memory Management: Implementing aggressive memory sharing to keep the runtime footprint as small as possible.
Evaluation
Datasets and Setup
The models were evaluated on standard industry benchmarks: WMT14 English-German (En-De) with 4.5M sentence pairs and WMT14 English-French (En-Fr) with 35M sentence pairs. Performance was measured on a Google Pixel 4 smartphone.
Key Results
MobileNMT was compared against standard Transformer-Big and Base models running on TFLite:
- Latency and Footprint: On the Pixel 4, the MobileNMT-10MB model achieved a latency of 27.3ms and used only 14.9MB of RAM. This represents a 47x speedup and a 99.5% reduction in memory compared to the Transformer-Big baseline.
- Translation Quality: Despite the massive reduction in size, the 10MB model achieved a BLEU score of 25.08 on En-De, retaining approximately 88.4% of the quality of the 213M-parameter Big model (28.36 BLEU).
- Efficiency: Compared to other specialized architectures like DeLighT or Lite Transformer, MobileNMT demonstrated a superior trade-off between computational cost (FLOPs) and actual translation accuracy.
Discussion and Future Work
A significant insight from this work is that "quantization-friendliness" should be a primary design goal for mobile models. Methods that reduce parameter counts but broaden weight distributions (like parameter sharing) can be counterproductive when moving to 8-bit inference.
Currently, the system is optimized for bilingual translation and ARM CPUs. Future iterations aim to explore multilingual translation—which requires greater model capacity—and extend support to dedicated AI accelerators like NPUs (Neural Processing Units). While the final model is tiny, the training process still requires a full-sized Teacher model, highlighting a continued need for high-performance computing during the development phase.
Conclusion
MobileNMT demonstrates that through the co-design of model architecture and inference engines, it is possible to bring high-quality neural machine translation to the most resource-constrained environments. By prioritizing INT8 compatibility and asymmetric depth, MobileNMT achieves sub-30ms translation in under 15MB, setting a new benchmark for on-device AI.
Summary updated