TranslateLocally: Blazing-fast translation running on the local CPU

Nikolay Bogoychev, Jelmer Van der Linde, Kenneth Heafield

TL;DR

translateLocally is an open-source, cross-platform desktop application that brings fast, high-quality neural machine translation directly to local consumer CPUs, completely eliminating the privacy risks of cloud-based alternatives. By combining heavy optimization, knowledge distillation, and 8-bit quantization, it achieves translation speeds up to 100x faster than competing local tools while maintaining or exceeding their translation quality—all with a model size of just 15MB.

What Problem Does This Paper Address?

Modern Neural Machine Translation (NMT) is highly accurate but computationally expensive, which has pushed most services into the cloud (e.g., Google Translate, Microsoft Translator). However, sending data to cloud providers introduces significant privacy risks, including data breaches, data logging for advertising, or third-party contractor reviews. For individuals, corporations, and governments with strict confidentiality requirements, cloud translation is often banned or prohibitively expensive.

While local translation tools exist to protect privacy, they traditionally suffer from slow performance, lack of cross-platform support, or massive model sizes that are impractical for standard consumer hardware. The authors address this gap by creating a local translation system that runs efficiently on standard desktop and laptop CPUs—even hardware that is a decade old—without requiring an internet connection or a dedicated GPU.

Methods

The authors developed translateLocally using the open-source Marian machine translation toolkit as a base, heavily optimizing its core engine for consumer CPUs. Their approach relies on several key optimization techniques:

  • Knowledge Distillation: They trained small, highly efficient "student" models from large, complex "teacher" models to compress the network architecture without severe quality loss.
  • 8-bit Integer Quantization: Standard float32 arithmetic is replaced with efficient 8-bit integer operations, drastically reducing CPU overhead and memory footprint.
  • Lexical Shortlists: The system dynamically restricts the vocabulary size of the output layer based on the input text, minimizing unnecessary computations.
  • Simplified Simple Recurrent Units (SSRU): Used to accelerate the decoding process.
  • Hardware-Specific Vectorization: The underlying matrix multiplication library dynamically dispatches optimized instruction sets (SSSE3, AVX2, AVX512BW, and AVX512VNNI) based on the host CPU's capabilities.
  • Streamlined Architecture: A lightweight C++ wrapper handles sentence splitting and formatting preservation, paired with a minimalist Qt5/Qt6 graphical user interface.

Datasets

The system was trained and evaluated using several standard and custom datasets:

  • WNGT 2020 Efficiency Shared Task Dataset: A million-sentence test set used to benchmark translation speed across various consumer CPU architectures.
  • WMT Test Sets (up to WMT19): Used to evaluate translation quality (BLEU scores) across the majority of the supported language pairs.
  • Self-Crawled TED Talks Test Set: Specifically introduced and used to evaluate translation quality for Icelandic (is-en) and Norwegian (nn-en, nb-en) language pairs.

Key Insights and Ideas

  • Efficiency Doesn't Require Sacrificing Quality: Through student-teacher distillation and 8-bit quantization, the authors shrank their English-German model to just 15MB while actually improving the BLEU score compared to much larger competing models (e.g., OPUS-CAT's 275MB model).
  • Local Latency Can Match Cloud Performance: The system is optimized enough that translations update fluidly as the user types, achieving a local latency comparable to network ping times to cloud servers.
  • Architectural Bottlenecks Matter: Competing tools like OPUS-CAT and Argos Translate use similar underlying translation frameworks but are bottlenecked by unoptimized engines, single-threaded constraints, or slow GUI text handling. translateLocally bypasses this by utilizing a custom C++ wrapper and optimized memory mapping.

Key Results

The authors evaluated translateLocally against two prominent local translation tools: Argos Translate and OPUS-CAT MT Engine.

  • Translation Quality: On the WMT19 English-German benchmark, translateLocally achieved a BLEU score of 41.8, outperforming OPUS-CAT (40.8) and Argos Translate (34.9).
  • Speed vs. Argos Translate: Tested on a 2019 MacBook Pro (8 threads) translating 40k tokens, translateLocally achieved 7,350 words per second (WPS) taking 40 seconds, compared to Argos Translate's 76 WPS taking 4,378 seconds (~100x speedup).
  • Speed vs. OPUS-CAT: In a single-threaded Windows environment, translateLocally achieved 1,250 WPS (34s) compared to OPUS-CAT's 12 WPS (3,363s), representing another ~100x speedup.
  • Hardware Versatility: The software runs efficiently even on a 2012 dual-core laptop CPU, maintaining a usable throughput of 1,066 WPS.

Limitations and Future Work

  • Language Support: The initial release only supports 10 language pairs, though the authors plan to continuously add more models to the project.
  • CAT Integration: Unlike OPUS-CAT, translateLocally does not currently support plugins for professional Computer-Assisted Translation (CAT) software used by professional translators.
  • Kernel Optimization: While matrix multiplication dynamically dispatches vectorized instructions, other kernels (like activation functions) are not yet multi-versioned and require explicit compilation to maximize performance on specific instruction sets.
  • Batching vs. Sentence Latency: While throughput is incredibly high for large documents due to batching, users translating only a few sentences will experience lower overall throughput (though individual sentence latency remains low).

Summary updated

Abstract

Every day, millions of people sacrifice their privacy and browsing habits in exchange for online machine translation. Companies and governments with confidentiality requirements often ban online translation or pay a premium to disable logging. To bring control back to the end user and demonstrate speed, we developed translateLocally. Running locally on a desktop or laptop CPU, translateLocally delivers cloud-like translation speed and quality even on 10 year old hardware. The open-source software is based on Marian and runs on Linux, Windows, and macOS.
BibTeX
@inproceedings{bogoychev2021translatelocally,
  author = {Nikolay Bogoychev and Jelmer Van der Linde and Kenneth Heafield},
  title = {{TranslateLocally: Blazing-fast translation running on the local CPU}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
}
Figure