The Pipeline System of ASR and NLU with MLM-based Data Augmentation toward STOP Low-resource Challenge

Hayato Futami, Jessica Huynh, Siddhant Arora, Shih-Lun Wu, Yosuke Kashiwagi, Yifan Peng, Brian Yan, Emiru Tsunoo, Shinji Watanabe

Introduction

Spoken Language Understanding (SLU) is a critical component of modern voice-controlled systems, tasked with converting raw audio signals into structured semantic representations. This process typically involves identifying a user's intent (e.g., get_weather) and extracting specific slots (e.g., location: sydney). While end-to-end (E2E) models have gained popularity for their simplicity, they often struggle in "low-resource" scenarios where labeled training data for specific domains is extremely scarce.

This paper details the winning approach for Track 3 of the ICASSP 2023 Spoken Language Understanding Grand Challenge, which focused on the Spoken Task-Oriented Semantic Parsing (STOP) dataset. The challenge required systems to adapt to new domains—specifically "reminder" and "weather"—using as few as 162 training examples. The researchers from Sony and Carnegie Mellon University demonstrated that a robust pipeline approach, enhanced by targeted data augmentation and retrieval-based context, significantly outperforms E2E alternatives in these data-constrained environments.

System Architecture: The Pipeline Approach

Rather than using a single model to map audio directly to semantics, the authors employed a modular pipeline. This strategy allows the system to leverage massive, disparate datasets for speech recognition and natural language understanding independently.

1. Automatic Speech Recognition (ASR)

The ASR component utilizes OpenAI’s Whisper model. To adapt Whisper to the specific vocabulary and acoustic conditions of the STOP challenge, the authors fine-tuned the model while employing several regularization techniques to prevent overfitting on the small target datasets. These included:

  • Upsampling: The low-resource domain data was upsampled by a factor of 20.
  • SpecAugment: Applying frequency and time masking to the audio features.
  • Label Smoothing: Improving the model's generalization by preventing it from becoming overconfident in its predictions.

2. Natural Language Understanding (NLU)

The NLU component is based on BART-large, a sequence-to-sequence transformer. The fine-tuning process followed a two-stage curriculum: first, the model was trained on the entirety of the STOP challenge data to learn general semantic parsing; second, it was specifically fine-tuned on the low-resource target domains to specialize in those intents and slots.

Data Augmentation and Contextual Retrieval

To overcome the lack of training samples, the authors introduced two primary innovations: MLM-based augmentation and Retrieval-Augmented BART (RA-BART).

MLM-Based Data Augmentation

The researchers used a Masked Language Model (MLM)—specifically BERT—to generate synthetic training data. By masking tokens in existing sentences (e.g., "how's the weather in [MASK]"), BERT could predict plausible alternatives (e.g., "London"). If the masked word corresponded to a slot value in the semantic parse, the target label was automatically updated to match the new word. To ensure the quality of this synthetic data, a filtering step was implemented: only samples that could be correctly parsed by an NLU model trained on the original ground-truth data were retained.

Retrieval-Augmented BART (RA-BART)

During the inference phase, the NLU model does not process the ASR transcript in isolation. Instead, the system uses TF-IDF similarity to retrieve the four most relevant examples from the training set. These "exemplars" are prepended to the input transcript, providing the model with immediate context and examples of how similar requests should be structured. This effectively transforms the task into a few-shot learning problem at runtime.

Experimental Results

The proposed system achieved 1st place in the STOP Challenge, demonstrating a massive performance gap over the baseline models.

Domain Exact Match (EM) Accuracy
Reminder 63.3%
Weather 75.0%
Average (Pipeline) 69.15%
Average (E2E Baseline) 31.08%

The results highlight the efficacy of the augmentation strategies. For the weather domain, the NLU accuracy on ground-truth text improved from 65.1% to 79.7% once MLM-based augmentation and retrieval context were added.

Discussion and Future Work

The success of this pipeline system underscores that for low-resource SLU, leveraging large-scale pre-trained models (Whisper and BART) is more effective than training E2E models from scratch. However, the authors noted several areas for improvement:

  • Error Propagation: Mistakes made by the ASR (e.g., phonetic misinterpretations) directly impact the NLU's ability to parse the intent correctly.
  • Computational Overhead: The use of large transformer models and a retrieval step increases the latency and computational requirements compared to smaller E2E models.
  • Synthetic Quality: While the MLM approach was successful, some generated samples were grammatically awkward, suggesting that more advanced generative models could further improve data quality.

In conclusion, the combination of a high-performance pipeline, intelligent data synthesis, and retrieval-based context provides a powerful framework for extending voice assistants into new domains with minimal human-labeled data.

Summary updated

Abstract

This paper describes our system for the low-resource domain adaptation track (Track 3) in Spoken Language Understanding Grand Challenge, which is a part of ICASSP Signal Processing Grand Challenge 2023. In the track, we adopt a pipeline approach of ASR and NLU. For ASR, we fine-tune Whisper for each domain with upsampling. For NLU, we fine-tune BART on all the Track3 data and then on low-resource domain data. We apply masked LM (MLM) -based data augmentation, where some of input tokens and corresponding target labels are replaced using MLM. We also apply a retrieval-based approach, where model input is augmented with similar training samples. As a result, we achieved exact match (EM) accuracy 63.3/75.0 (average: 69.15) for reminder/weather domain, and won the 1st place at the challenge.
BibTeX
@inproceedings{futami2023pipeline,
  author = {Hayato Futami and Jessica Huynh and Siddhant Arora and Shih-Lun Wu and Yosuke Kashiwagi and Yifan Peng and Brian Yan and Emiru Tsunoo and Shinji Watanabe},
  title = {{The Pipeline System of ASR and NLU with MLM-based Data Augmentation toward STOP Low-resource Challenge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096049},
}
Figure