Using Machine Translation to Localize Task Oriented NLG Output

Scott Roy, Cliff Brunk, Kyu-Young Kim, Justin Zhao, Markus Freitag, Mihir Kale, Gagan Bansal, Sidharth Mudgal, Chris VaranoarXiv 2021 · OpenReview

Abstract

One of the challenges in a task oriented natural language application like the Google Assistant, Siri, or Alexa is to localize the output to many languages. This paper explores doing this by applying machine translation to the English output. Using machine translation is very scalable, as it can work with any English output and can handle dynamic text, but otherwise the problem is a poor fit. The required quality bar is close to perfection, the range of sentences is extremely narrow, and the sentences are often very different than the ones in the machine translation training data. This combination of requirements is novel in the field of domain adaptation for machine translation. We are able to reach the required quality bar by building on existing ideas and adding new ones: finetuning on in-domain translations, adding sentences from the Web, adding semantic annotations, and using automatic error detection. The paper shares our approach and results, together with a distillation model to serve the translation models at scale.

Introduction

Localizing virtual assistants is a massive scaling challenge. Traditionally, developers rely on hand-written templates for every language, which is labor-intensive and fails to handle dynamic text well. While Neural Machine Translation (NMT) seems like an obvious alternative, it usually fails the "perfection" test required for user-facing products. General-purpose NMT models often struggle with domain mismatch, literal translations of entities (e.g., turning "Palo Alto" into "High Stick"), accuracy errors such as "hallucinations," and high latency during real-time inference.

However, virtual assistant output is "narrow and deep." By focusing on a limited range of intents (weather, timers, etc.), it is possible to reach a quality bar (97% accuracy) that general NMT cannot touch. This paper proposes a pipeline that leverages NMT not as the final product, but as a high-quality "labeler" for a faster, simpler architecture.

Methodology

The authors propose a multi-stage pipeline that moves from "high-quality but slow" models to "high-quality and fast" production models.

The multi-stage localization pipeline, from offline NMT training to a distilled data-to-text serving model.

Domain-Adapted NMT

The process starts with a standard NMT model finetuned using a mix of high-quality Web sentences (filtered by a Bayesian classifier) and back-translation. To make the most of a small set of human-translated "gold" examples, Contrastive Data Selection (CDS) is used to identify and weight similar sentences from a massive pool of Web data. Additionally, Named Entity Recognition (NER) marks spans (e.g., <location>) in the training data, teaching the model to treat these as special tokens rather than translatable words.

Accuracy Error Detection

To prevent factual errors, a specific classifier was built to catch hallucinations. This model was trained using "synthetic corruption": correct translations were modified by replacing words with others appearing in the same linguistic context. This created "smart" negative examples that forced the model to pay attention to factual accuracy.

The process of generating negative examples for error detection by replacing words with others found in similar linguistic contexts.

Model Distillation

Instead of serving the heavy NMT model, it is used offline to translate millions of synthetic examples. These examples then train a lightweight, RNN-based data-to-text model that maps structured data directly to the target language, bypassing the English intermediate step during live inference to ensure low latency.

Datasets and Experimental Setup

The research focused on eight low-resource Indic languages: Marathi, Bengali, Tamil, Telugu, Gujarati, Kannada, Malayalam, and Urdu. Training data consisted of approximately 200–300M parallel sentences per language, with a small finetuning set of 1k–5k in-domain sentences. Evaluation was conducted on a synthetic heldout set and a human-evaluated set to test generalization to unseen intents and domains.

Results

The improvements over a baseline NMT model were dramatic, especially for "unseen" data. BLEU scores on unseen values jumped from 21.19 to 62.78. Human evaluation showed the task-specific model achieved 97% accuracy, compared to 68% for the baseline. Furthermore, in languages like Marathi and Malayalam, the NMT-generated output was rated as more natural than human translations because the model avoided the "code-switching" (mixing English words) common in human-provided data.

Conclusion

While the system achieves high accuracy, limitations remain regarding the phonetic transliteration of rare names and handling social nuances like gender agreement and honorifics. Future work will explore moving toward a single unified model that combines the NMT and data-to-text steps to handle entity copying more reliably.

Summary updated

Abstract

One of the challenges in a task oriented natural language application like the Google Assistant, Siri, or Alexa is to localize the output to many languages. This paper explores doing this by applying machine translation to the English output. Using machine translation is very scalable, as it can work with any English output and can handle dynamic text, but otherwise the problem is a poor fit. The required quality bar is close to perfection, the range of sentences is extremely narrow, and the sentences are often very different than the ones in the machine translation training data. This combination of requirements is novel in the field of domain adaptation for machine translation. We are able to reach the required quality bar by building on existing ideas and adding new ones: finetuning on in-domain translations, adding sentences from the Web, adding semantic annotations, and using automatic error detection. The paper shares our approach and results, together with a distillation model to serve the translation models at scale.
BibTeX
@misc{roy2021using,
  author = {Scott Roy and Cliff Brunk and Kyu-Young Kim and Justin Zhao and Markus Freitag and Mihir Kale and Gagan Bansal and Sidharth Mudgal and Chris Varano},
  title = {{Using Machine Translation to Localize Task Oriented NLG Output}},
  year = {2021},
  eprint = {2107.04512},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.04512},
}
Figure