TL;DR
MobileNMT is a lightweight, on-device translation system capable of generating translations in under 30 milliseconds and taking up less than 15 MB of memory on a smartphone. By co-designing a quantization-friendly Transformer architecture (deep encoder, shallow decoder, reduced vocabulary) alongside a custom ARM-optimized INT8 inference engine, the authors achieve up to a 47× speedup and 99.5% memory reduction compared to standard Transformer baselines on mobile frameworks, with minimal drop in translation quality.
What Problem Does This Paper Address?
Deploying neural machine translation (NMT) directly onto mobile phones is critical for offline usability, low latency, and user privacy. However, state-of-the-art Transformer models typically require hundreds of megabytes (or gigabytes) of memory and billions of FLOPs.
Existing mobile runtimes (such as TensorFlow Lite) and compression techniques are often ill-suited for NMT. They either optimize for generic neural network metrics while hurting auto-regressive decoding speed, or struggle with the iterative loops required by sequential text generation. This paper tackles the challenge of end-to-end on-device translation under strict real-world constraints: sub-30ms latency, tiny memory footprints, and negligible BLEU degradation.
Methods
The authors adopt a hardware-model co-design approach:
- Architecture Principles for Edge Transformers:
- Vocabulary Pruning over Factorization: Instead of ALBERT-style matrix factorization for token embeddings, shrinking the Byte-Pair Encoding (BPE) vocabulary (e.g., to 8,000 merge operations) preserves parameter efficiency and BLEU much better.
- Narrow Width over Parameter Sharing: Reducing model width (hidden dimension) produces much smaller activation dynamic ranges than cross-layer parameter sharing, making the network substantially more quantization-friendly.
- Deep Encoder, Shallow Decoder: Since autoregressive decoder steps dominate latency on mobile processors, the architecture pairs a deep 12-layer encoder with a compact 2-layer decoder.
- Quantization & Training Strategy:
- Models are pre-trained via sequence-level knowledge distillation (KD) from a full-precision Transformer-base teacher.
- Dropout is completely removed, preventing underfitting on small networks and boosting BLEU by ~2 points.
- Stronger $L_2$ regularization (weight decay) is used alongside a higher learning rate to rein in large weight/activation outliers, preparing the model for Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).
- Uniform 8-bit integer (INT8) quantization is applied to linear layers in attention and FFN blocks, retaining biases and residual connections in full precision with a learnable fusion weight ($\gamma_i$).
- Dedicated ARM Inference Engine:
- General Matrix Multiplication (GEMM) accounts for over 80% of decoding runtime. The authors write custom SIMD assembly kernels on ARM, packing $4 \times 4$ blocks into contiguous memory chunks to maximize register and cache utilization.
- Memory operator fusion consolidates fine-grained operations between GEMM calls, sharing buffers so that the entire encoder needs only two 8-bit scratch buffers and one 32-bit buffer during execution.
Datasets
The authors evaluate MobileNMT on standard translation benchmarks:
- WMT14 English-to-German (En-De) (4.5M training pairs): Validated on
newstest-2013, tested onnewstest-2014. - WMT14 English-to-French (En-Fr) (35M training pairs): Validated on
newstest-2012 + 2013, tested onnewstest-2014.
Both language pairs employ shared subword vocabularies to maximize compression. No new datasets were introduced.
Key Insights and Ideas
- Parameter sharing is hostile to INT8 quantization: While weight sharing across layers keeps parameter counts low on paper, it expands the dynamic range of activations and weights across layers, causing significant quantization error during PTQ.
- Asymmetric depth is ideal for latency: Most decoding latency in NMT comes from sequential decoder steps. Shifting model depth into the encoder (which runs in parallel in a single shot) retains strong representation capacity while keeping token generation fast.
- Dropout hurts compact models: Regularization techniques designed for large over-parameterized models penalize capacity-starved edge models; dropping dropout yields an immediate ~2 BLEU improvement.
Key Results
Experiments run locally on a Google Pixel 4 smartphone (measuring length-30 translation sequences) show dramatic performance gains:
| Model | Size | Memory | Latency | En-De BLEU |
|---|---|---|---|---|
| Transformer-big (TFLite) | 872 MB | 2,886.6 MB | 1,281.5 ms | 28.36 |
| Transformer-base (TFLite) | 260 MB | 908.5 MB | 332.3 ms | 27.40 |
| MobileNMT-20MB (Ours) | 20 MB | 26.0 MB | 46.3 ms | 27.09 |
| MobileNMT-10MB (Ours) | 10 MB | 14.9 MB | 27.3 ms | 25.08 |
- Compared to Transformer-big, MobileNMT-10MB is 47.0× faster, uses 99.5% less memory, and compresses the model by 87.2×, while retaining 88.4% of the original BLEU score (25.08 vs. 28.36).
- On WMT14 En-Fr, MobileNMT-10MB achieves 36.00 BLEU at 28.9 ms per sentence.
- Weight quantization down to 4-bit remains robust (e.g., 25.43 BLEU at 5 MB for MobileNMT-10MB), though 3-bit and 2-bit weights show sharp quality degradation.
Limitations and Future Work
- Bilingual Focus: Experiments are currently limited to bilingual translation; scaling the approach to multilingual models without excessive parameter bloat remains an open challenge.
- Distillation Overhead: Achieving high BLEU scores at 10M–20M parameters relies heavily on knowledge distillation, introducing additional compute overhead during the offline training stage.
- Hardware Support: The custom inference engine is currently tailored specifically to ARM CPUs; extending optimizations to on-device Neural Processing Units (NPUs) and mobile GPUs is planned for future work.
Summary updated