Abstract
Scaling voice assistants to new languages is often hindered by a lack of manually labeled data. This paper presents an offline "label projection" pipeline that leverages machine translation (MT) and high-performing English Natural Language Understanding (NLU) models to automatically annotate unlabeled data in target languages like German. Experimental results show that this approach can reduce the need for manual labeling by up to 90% while significantly improving performance in data-scarce scenarios.
Introduction
The rapid expansion of voice-controlled AI necessitates high-quality NLU models across dozens of languages. While English benefits from vast amounts of "gold-standard" manual annotations, low-resource languages often lack sufficient data for training robust intent classifiers and slot taggers. Common strategies like "translate-test" introduce unacceptable latency in production. This work focuses on "feature improvement" through an offline data augmentation pipeline that utilizes millions of unlabeled utterances to boost model performance without increasing runtime latency.
Methodology
The core of the proposed system is a multi-stage pipeline designed to transfer semantic knowledge from a source language (English) to a target language (German).
- Translation: Unlabeled German utterances are translated into English using a statistical MT system.
- Source Annotation: A mature English NLU model predicts the intent and identifies slots within the translated text.
- Label Projection: Using word-alignment information from the MT step, labels are mapped back to the original German tokens.
- Confidence Filtering: To mitigate noise, a confidence threshold (e.g., 0.5) is applied to the NLU predictions, filtering out low-certainty "silver-standard" labels.
Experimental Setup
The researchers evaluated the pipeline using approximately 6.4 million unlabeled German utterances from a commercial dialog system. The evaluation spanned five functional domains: DailyBriefing, PlayMusic, SendMessage, SmartHome, and SetNotification. To simulate various stages of feature maturity, models were trained on varying amounts of manual "gold" data, ranging from 100 to 10,000 samples, augmented with the auto-labeled "silver" data.
Results and Discussion
The results demonstrate that machine-generated labels, while imperfect, provide a powerful signal for model training. Intrinsic evaluation showed that 56.35% of auto-labeled utterances perfectly matched human annotations, with over 81% achieving correct intent classification.
The most significant performance gains occurred in low-data regimes. For instance, in the SmartHome domain with only 100 manual samples, the addition of auto-labeled data reduced the Semantic Error Rate (SemER) by 74.34 points.
A key finding is the "90% rule": models trained on 100 manual samples plus auto-labeled data often outperformed those trained on 1,000 manual samples alone. This represents a massive cost saving for industrial applications. However, as the volume of manual data increases (e.g., beyond 10,000 samples), the relative benefit of auto-labeled data diminishes, and the noise from translation or alignment errors can eventually outweigh the benefits of the extra data.
Conclusion
This research provides a scalable framework for improving NLU in low-resource languages by "borrowing" labels from high-resource counterparts. By effectively utilizing unlabeled data through an offline translation and projection pipeline, developers can achieve target performance levels with a fraction of the manual labeling effort, significantly accelerating the deployment of global voice AI features.
Summary updated