Abstract
One of the challenges in a task oriented natural language application like the Google Assistant, Siri, or Alexa is to localize the output to many languages. This paper explores doing this by applying machine translation to the English output. Using machine translation is very scalable, as it can work with any English output and can handle dynamic text, but otherwise the problem is a poor fit. The required quality bar is close to perfection, the range of sentences is extremely narrow, and the sentences are often very different than the ones in the machine translation training data. This combination of requirements is novel in the field of domain adaptation for machine translation. We are able to reach the required quality bar by building on existing ideas and adding new ones: finetuning on in-domain translations, adding sentences from the Web, adding semantic annotations, and using automatic error detection. The paper shares our approach and results, together with a distillation model to serve the translation models at scale.
Introduction
Localizing virtual assistants is a massive scaling challenge. Traditionally, developers rely on hand-written templates for every language, which is labor-intensive and fails to handle dynamic text well. While Neural Machine Translation (NMT) seems like an obvious alternative, it usually fails the "perfection" test required for user-facing products. General-purpose NMT models often struggle with domain mismatch, literal translations of entities (e.g., turning "Palo Alto" into "High Stick"), accuracy errors such as "hallucinations," and high latency during real-time inference.
However, virtual assistant output is "narrow and deep." By focusing on a limited range of intents (weather, timers, etc.), it is possible to reach a quality bar (97% accuracy) that general NMT cannot touch. This paper proposes a pipeline that leverages NMT not as the final product, but as a high-quality "labeler" for a faster, simpler architecture.
Methodology
The authors propose a multi-stage pipeline that moves from "high-quality but slow" models to "high-quality and fast" production models.
Domain-Adapted NMT
The process starts with a standard NMT model finetuned using a mix of high-quality Web sentences (filtered by a Bayesian classifier) and back-translation. To make the most of a small set of human-translated "gold" examples, Contrastive Data Selection (CDS) is used to identify and weight similar sentences from a massive pool of Web data. Additionally, Named Entity Recognition (NER) marks spans (e.g., <location>) in the training data, teaching the model to treat these as special tokens rather than translatable words.
Accuracy Error Detection
To prevent factual errors, a specific classifier was built to catch hallucinations. This model was trained using "synthetic corruption": correct translations were modified by replacing words with others appearing in the same linguistic context. This created "smart" negative examples that forced the model to pay attention to factual accuracy.
Model Distillation
Instead of serving the heavy NMT model, it is used offline to translate millions of synthetic examples. These examples then train a lightweight, RNN-based data-to-text model that maps structured data directly to the target language, bypassing the English intermediate step during live inference to ensure low latency.
Datasets and Experimental Setup
The research focused on eight low-resource Indic languages: Marathi, Bengali, Tamil, Telugu, Gujarati, Kannada, Malayalam, and Urdu. Training data consisted of approximately 200–300M parallel sentences per language, with a small finetuning set of 1k–5k in-domain sentences. Evaluation was conducted on a synthetic heldout set and a human-evaluated set to test generalization to unseen intents and domains.
Results
The improvements over a baseline NMT model were dramatic, especially for "unseen" data. BLEU scores on unseen values jumped from 21.19 to 62.78. Human evaluation showed the task-specific model achieved 97% accuracy, compared to 68% for the baseline. Furthermore, in languages like Marathi and Malayalam, the NMT-generated output was rated as more natural than human translations because the model avoided the "code-switching" (mixing English words) common in human-provided data.
Conclusion
While the system achieves high accuracy, limitations remain regarding the phonetic transliteration of rare names and handling social nuances like gender agreement and honorifics. Future work will explore moving toward a single unified model that combines the NMT and data-to-text steps to handle entity copying more reliably.
Summary updated