Continuous Model Improvement for Language Understanding with Machine Translation

Abdalghani Abujabal, Claudio Delli Bovi, Sungho Ryu, Turan Gojayev, Fabian Triefenbach, Yannick VersleyNAACL 2021 · DOI

Abstract

Scaling voice assistants to new languages is often hindered by a lack of manually labeled data. This paper presents an offline "label projection" pipeline that leverages machine translation (MT) and high-performing English Natural Language Understanding (NLU) models to automatically annotate unlabeled data in target languages like German. Experimental results show that this approach can reduce the need for manual labeling by up to 90% while significantly improving performance in data-scarce scenarios.

Introduction

The rapid expansion of voice-controlled AI necessitates high-quality NLU models across dozens of languages. While English benefits from vast amounts of "gold-standard" manual annotations, low-resource languages often lack sufficient data for training robust intent classifiers and slot taggers. Common strategies like "translate-test" introduce unacceptable latency in production. This work focuses on "feature improvement" through an offline data augmentation pipeline that utilizes millions of unlabeled utterances to boost model performance without increasing runtime latency.

Methodology

The core of the proposed system is a multi-stage pipeline designed to transfer semantic knowledge from a source language (English) to a target language (German).

  1. Translation: Unlabeled German utterances are translated into English using a statistical MT system.
  2. Source Annotation: A mature English NLU model predicts the intent and identifies slots within the translated text.
  3. Label Projection: Using word-alignment information from the MT step, labels are mapped back to the original German tokens.
  4. Confidence Filtering: To mitigate noise, a confidence threshold (e.g., 0.5) is applied to the NLU predictions, filtering out low-certainty "silver-standard" labels.
The label projection pipeline: German utterances are translated, annotated by an English NLU system, and labels are projected back using word alignment.

Experimental Setup

The researchers evaluated the pipeline using approximately 6.4 million unlabeled German utterances from a commercial dialog system. The evaluation spanned five functional domains: DailyBriefing, PlayMusic, SendMessage, SmartHome, and SetNotification. To simulate various stages of feature maturity, models were trained on varying amounts of manual "gold" data, ranging from 100 to 10,000 samples, augmented with the auto-labeled "silver" data.

Results and Discussion

The results demonstrate that machine-generated labels, while imperfect, provide a powerful signal for model training. Intrinsic evaluation showed that 56.35% of auto-labeled utterances perfectly matched human annotations, with over 81% achieving correct intent classification.

The most significant performance gains occurred in low-data regimes. For instance, in the SmartHome domain with only 100 manual samples, the addition of auto-labeled data reduced the Semantic Error Rate (SemER) by 74.34 points.

Reduction in Semantic Error Rate (SemER) across different data splits and features. Negative values indicate performance improvement.

A key finding is the "90% rule": models trained on 100 manual samples plus auto-labeled data often outperformed those trained on 1,000 manual samples alone. This represents a massive cost saving for industrial applications. However, as the volume of manual data increases (e.g., beyond 10,000 samples), the relative benefit of auto-labeled data diminishes, and the noise from translation or alignment errors can eventually outweigh the benefits of the extra data.

Conclusion

This research provides a scalable framework for improving NLU in low-resource languages by "borrowing" labels from high-resource counterparts. By effectively utilizing unlabeled data through an offline translation and projection pipeline, developers can achieve target performance levels with a fraction of the manual labeling effort, significantly accelerating the deployment of global voice AI features.

Summary updated

BibTeX
@inproceedings{abujabal2021continuous,
  author = {Abdalghani Abujabal and Claudio Delli Bovi and Sungho Ryu and Turan Gojayev and Fabian Triefenbach and Yannick Versley},
  title = {{Continuous Model Improvement for Language Understanding with Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-industry.8},
}
Figure