Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding

Yingmei Guo, Linjun Shou, Jian Pei, Ming Gong, Mingxing Xu, Zhiyong Wu, Daxin Jiang

Abstract

Scaling Spoken Language Understanding (SLU) systems to multiple languages is hindered by the high cost of producing labeled data. While data augmentation techniques like machine translation and generative modeling can synthesize training data, they often introduce noise that degrades model performance. This paper proposes a novel denoising framework that utilizes multiple models trained on diverse noisy datasets to supervise each other. Through strategies such as instance relabeling, co-training, and instance re-weighting, the proposed method achieves a new state-of-the-art on major benchmarks, improving performance by over 4 percentage points.

Introduction

Spoken Language Understanding (SLU) is a core component of modern voice assistants, encompassing intent detection and slot tagging. Despite its success in English, extending these capabilities to other languages remains a challenge due to the lack of high-quality labeled datasets. Researchers typically rely on zero-resource cross-lingual transfer, where models are trained on English data and augmented with synthetic target-language data. However, synthetic data is inherently noisy—translations may be inaccurate, and generative models can hallucinate. This noise often leads to poor generalization in the target language.

Methodology

The proposed framework addresses the noise problem by treating it as a manageable factor through multi-perspective learning. The approach consists of two primary stages:

1. Data Augmentation

Two distinct noisy datasets are generated to provide diverse perspectives:

  • Machine Translation ($D_{trans}$): English data is translated using machine translation services, with word alignments established via tools like Giza++.
  • Generative Modeling ($D_{gen}$): A generative model (e.g., mBART) is fine-tuned to produce new utterances based on specific dialog acts, effectively "dreaming up" variations.

2. Denoising Training

Instead of a single model, an ensemble of $K$ models is trained. During a "Relabeling Stage," three key strategies are employed to filter noise:

  • Instance Relabeling: The ensemble predicts pseudo-labels for noisy data. If there is strong consensus that differs from the synthetic label, the label is updated.
  • Co-Training: Models act as teachers for one another. A model identifies high-confidence (low-loss) examples to serve as training data for its peers.
  • Instance Re-weighting: The variance between model predictions is used to identify uncertain examples, which are then down-weighted during training.
Overview of the proposed framework, illustrating the data augmentation pipeline and the iterative denoising training loop.

Experimental Setup

The framework was evaluated on two multilingual SLU benchmarks: SNIPS (English, Spanish, Thai) and MTOP (English, Spanish, French, German, Hindi, Thai). The experiments followed a zero-shot setup, where models were trained only on English and synthetic data, without any human-labeled target-language examples.

Results and Discussion

The proposed method significantly outperformed existing baselines, setting a new state-of-the-art for zero-shot cross-lingual SLU. On the SNIPS dataset, the method improved the F1/Intent Accuracy by 3.05 percentage points. On the more complex MTOP dataset, it achieved a 4.24 percentage point improvement in Exact Match Accuracy.

Main experimental results on SNIPS and MTOP datasets, comparing the proposed denoising framework against various baselines.

Qualitative analysis confirmed that the denoising module effectively corrected intent misclassifications and refined slot boundaries that were previously corrupted by translation errors. Furthermore, the independence of noise sources (translation vs. generation) proved crucial, as it allowed the models to effectively cross-verify each other's predictions.

Conclusion and Future Work

This work demonstrates that "crowd-intelligence" from multiple models can effectively mitigate the negative impact of noisy synthetic data in cross-lingual SLU. While the approach is computationally more intensive than single-model training, the performance gains are substantial. Future research will explore the applicability of this denoising framework to other cross-lingual tasks such as Named Entity Recognition and Question Answering, as well as optimizing the manual tuning required for co-training filtering rates.

Summary updated

BibTeX
@inproceedings{guo2021learning,
  author = {Yingmei Guo and Linjun Shou and Jian Pei and Ming Gong and Mingxing Xu and Zhiyong Wu and Daxin Jiang},
  title = {{Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.259},
}
Figure