MobileNMT: Enabling Translation in 15MB and 30ms

Ye Lin, Xiaohui Wang, Zhexi Zhang, Mingxuan Wang, Tong Xiao, Jingbo Zhu

TL;DR

MobileNMT is a lightweight, on-device translation system capable of generating translations in under 30 milliseconds and taking up less than 15 MB of memory on a smartphone. By co-designing a quantization-friendly Transformer architecture (deep encoder, shallow decoder, reduced vocabulary) alongside a custom ARM-optimized INT8 inference engine, the authors achieve up to a 47× speedup and 99.5% memory reduction compared to standard Transformer baselines on mobile frameworks, with minimal drop in translation quality.

What Problem Does This Paper Address?

Deploying neural machine translation (NMT) directly onto mobile phones is critical for offline usability, low latency, and user privacy. However, state-of-the-art Transformer models typically require hundreds of megabytes (or gigabytes) of memory and billions of FLOPs.

Existing mobile runtimes (such as TensorFlow Lite) and compression techniques are often ill-suited for NMT. They either optimize for generic neural network metrics while hurting auto-regressive decoding speed, or struggle with the iterative loops required by sequential text generation. This paper tackles the challenge of end-to-end on-device translation under strict real-world constraints: sub-30ms latency, tiny memory footprints, and negligible BLEU degradation.

Methods

The authors adopt a hardware-model co-design approach:

  1. Architecture Principles for Edge Transformers:
  2. Vocabulary Pruning over Factorization: Instead of ALBERT-style matrix factorization for token embeddings, shrinking the Byte-Pair Encoding (BPE) vocabulary (e.g., to 8,000 merge operations) preserves parameter efficiency and BLEU much better.
  3. Narrow Width over Parameter Sharing: Reducing model width (hidden dimension) produces much smaller activation dynamic ranges than cross-layer parameter sharing, making the network substantially more quantization-friendly.
  4. Deep Encoder, Shallow Decoder: Since autoregressive decoder steps dominate latency on mobile processors, the architecture pairs a deep 12-layer encoder with a compact 2-layer decoder.
  5. Quantization & Training Strategy:
  6. Models are pre-trained via sequence-level knowledge distillation (KD) from a full-precision Transformer-base teacher.
  7. Dropout is completely removed, preventing underfitting on small networks and boosting BLEU by ~2 points.
  8. Stronger $L_2$ regularization (weight decay) is used alongside a higher learning rate to rein in large weight/activation outliers, preparing the model for Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).
  9. Uniform 8-bit integer (INT8) quantization is applied to linear layers in attention and FFN blocks, retaining biases and residual connections in full precision with a learnable fusion weight ($\gamma_i$).
  10. Dedicated ARM Inference Engine:
  11. General Matrix Multiplication (GEMM) accounts for over 80% of decoding runtime. The authors write custom SIMD assembly kernels on ARM, packing $4 \times 4$ blocks into contiguous memory chunks to maximize register and cache utilization.
  12. Memory operator fusion consolidates fine-grained operations between GEMM calls, sharing buffers so that the entire encoder needs only two 8-bit scratch buffers and one 32-bit buffer during execution.

Datasets

The authors evaluate MobileNMT on standard translation benchmarks:

  • WMT14 English-to-German (En-De) (4.5M training pairs): Validated on newstest-2013, tested on newstest-2014.
  • WMT14 English-to-French (En-Fr) (35M training pairs): Validated on newstest-2012 + 2013, tested on newstest-2014.

Both language pairs employ shared subword vocabularies to maximize compression. No new datasets were introduced.

Key Insights and Ideas

  • Parameter sharing is hostile to INT8 quantization: While weight sharing across layers keeps parameter counts low on paper, it expands the dynamic range of activations and weights across layers, causing significant quantization error during PTQ.
  • Asymmetric depth is ideal for latency: Most decoding latency in NMT comes from sequential decoder steps. Shifting model depth into the encoder (which runs in parallel in a single shot) retains strong representation capacity while keeping token generation fast.
  • Dropout hurts compact models: Regularization techniques designed for large over-parameterized models penalize capacity-starved edge models; dropping dropout yields an immediate ~2 BLEU improvement.

Key Results

Experiments run locally on a Google Pixel 4 smartphone (measuring length-30 translation sequences) show dramatic performance gains:

Model Size Memory Latency En-De BLEU
Transformer-big (TFLite) 872 MB 2,886.6 MB 1,281.5 ms 28.36
Transformer-base (TFLite) 260 MB 908.5 MB 332.3 ms 27.40
MobileNMT-20MB (Ours) 20 MB 26.0 MB 46.3 ms 27.09
MobileNMT-10MB (Ours) 10 MB 14.9 MB 27.3 ms 25.08
  • Compared to Transformer-big, MobileNMT-10MB is 47.0× faster, uses 99.5% less memory, and compresses the model by 87.2×, while retaining 88.4% of the original BLEU score (25.08 vs. 28.36).
  • On WMT14 En-Fr, MobileNMT-10MB achieves 36.00 BLEU at 28.9 ms per sentence.
  • Weight quantization down to 4-bit remains robust (e.g., 25.43 BLEU at 5 MB for MobileNMT-10MB), though 3-bit and 2-bit weights show sharp quality degradation.

Limitations and Future Work

  • Bilingual Focus: Experiments are currently limited to bilingual translation; scaling the approach to multilingual models without excessive parameter bloat remains an open challenge.
  • Distillation Overhead: Achieving high BLEU scores at 10M–20M parameters relies heavily on knowledge distillation, introducing additional compute overhead during the offline training stage.
  • Hardware Support: The custom inference engine is currently tailored specifically to ARM CPUs; extending optimizations to on-device Neural Processing Units (NPUs) and mobile GPUs is planned for future work.

Summary updated

Abstract

Deploying NMT models on mobile devices is essential for privacy, low latency, and offline scenarios. For high model capacity, NMT models are rather large. Running these models on devices is challenging with limited storage, memory, computation, and power consumption. Existing work either only focuses on a single metric such as FLOPs or general engine which is not good at auto-regressive decoding. In this paper, we present MobileNMT, a system that can translate in 15MB and 30ms on devices. We propose a series of principles for model compression when combined with quantization. Further, we implement an engine that is friendly to INT8 and decoding. With the co-design of model and engine, compared with the existing system, we speed up 47.0x and save 99.5% of memory with only 11.6% loss of BLEU. Our code will be publicly available after the anonymity period.
BibTeX
@inproceedings{lin2023mobilenmt,
  author = {Ye Lin and Xiaohui Wang and Zhexi Zhang and Mingxuan Wang and Tong Xiao and Jingbo Zhu},
  title = {{MobileNMT: Enabling Translation in 15MB and 30ms}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=aIiJpr29wN},
}

Loading related content...

Figure