TL;DR
This paper introduces TranslateLocally, an open-source desktop application that brings fast, high-quality machine translation directly to your local CPU, bypassing cloud services. It achieves this by optimizing the Marian neural machine translation toolkit for CPU performance and model size, offering a privacy-preserving and responsive alternative to cloud-based translation.
What Problem Does This Paper Address?
Modern neural machine translation (NMT) systems, while powerful, are computationally expensive. This has led to their widespread deployment in the cloud by providers like Google, Microsoft, and Amazon. However, using cloud-based translation services raises significant privacy concerns. Users lose control over their data, which could be exposed through terms of service ambiguities, contractor access, or data breaches. Furthermore, organizations with strict confidentiality requirements often ban cloud translation or incur substantial costs to disable logging. This paper addresses the need for a fast, high-quality, and privacy-preserving machine translation solution that can run entirely on a user's local hardware, without relying on internet connectivity after the initial model download.
Methods
The core of TranslateLocally is built upon the Marian machine translation toolkit, a highly optimized NMT framework. The authors focused on several key areas to achieve blazing-fast CPU performance and reduced model size:
- CPU Optimization: They heavily optimized Marian for CPU execution, moving away from GPU reliance. This involved techniques like 8-bit integer arithmetic for model compression and inference, and binary lexical shortlists to streamline model loading.
- Model Compression: To make models small enough for local download and efficient execution, they employed knowledge distillation and lexical shortlists. This process transfers knowledge from larger, more complex models to smaller ones, significantly reducing their size while aiming to preserve translation quality.
- Streamlined Loading: They introduced streamlined binary model loading to the Marian codebase, reducing the time it takes to start translating.
- Sentence Splitting and Formatting Preservation: A C++ wrapper was developed to handle sentence splitting and preserve original formatting, crucial for translating larger documents.
- Qt-based GUI: A user-friendly graphical interface was developed using the Qt5 framework, ensuring cross-platform compatibility (Linux, Windows, macOS).
Datasets
The paper mentions using the WNGT 2020 efficiency shared task test set, which consists of one million sentences, for evaluating translation speed. For quality evaluation (BLEU scores), they used WMT test sets up to WMT19. For Icelandic and Norwegian language pairs, they used a self-crawled TED Talks test set.
Key Insights and Ideas
- CPU-centric NMT is Viable: The paper demonstrates that with careful optimization, NMT can achieve cloud-like translation speeds and quality directly on consumer-grade CPUs, even on older hardware.
- Privacy as a Primary Driver: The motivation for
TranslateLocallyis strongly rooted in user privacy, offering a compelling alternative for individuals and organizations concerned about data exposure. - Model Size vs. Performance Trade-off: Through techniques like knowledge distillation and 8-bit quantization, they achieved a remarkable balance between model size (as small as 15MB) and translation speed/quality.
- User Experience Matters: The inclusion of a responsive GUI that updates translations as the user types, along with features like model management and side-by-side view, significantly enhances usability.
Key Results
- Speed:
TranslateLocallyachieves translation speeds comparable to cloud services, with latency as low as ping times. On a MacBook Pro 16" 2019, it delivered 7350 WPS (words per second) for English-German translation. - Quality: The "tiny11" English-German model achieved a BLEU score of 41.8 on the WMT19 test set, outperforming Argos Translate (34.9 BLEU) and being comparable to OPUS-CAT (40.8 BLEU).
- Model Size: The "tiny11" English-German model is only 15MB, significantly smaller than OPUS-CAT's 275MB and Argos Translate's 87MB models.
- Performance Comparison:
TranslateLocallywas found to be approximately 100 times faster than Argos Translate and OPUS-CAT in their respective comparisons, even when OPUS-CAT was forced into single-threaded mode.
Limitations and Future Work
- Language Pairs: While the initial release includes 10 language pairs, the authors aim to continuously add more.
- CAT Tool Integration:
TranslateLocallydoes not currently support integration with professional Computer-Assisted Translation (CAT) tools, unlike OPUS-CAT. - Model Training: While users can train their own models, the process requires technical expertise. The paper mentions that the "student architecture" coupled with 8-bit integer compression delivers the smallest model size and highest BLEU score.
- Future Work: The authors plan to ship a version with models included, making it easier for users to set up a system without an internet connection. They are also building and adding new models to the project.
Summary updated