Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

Cheng-I Lai, Yung-Sung Chuang, Hung-yi Lee, Shang-wen Li, James R. Glass

Abstract: Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.

TL;DR

This paper introduces a robust framework for Spoken Language Understanding (SLU) that extracts semantics directly from audio by combining pretrained speech recognition (ASR) and language models (BERT). By leveraging semi-supervised pretraining on large unlabeled datasets, the model achieves performance on par with systems using perfect "oracle" text, even in the presence of significant environmental noise.

What Problem Does This Paper Address?

Spoken Language Understanding (SLU) is the "brain" behind virtual assistants, responsible for identifying a user's intent (e.g., "book a flight") and extracting specific slots (e.g., "from Boston to Denver").

Historically, this field has faced three major hurdles:

  1. Error Propagation: Most systems are "cascaded"—an ASR engine converts speech to text, and a Natural Language Understanding (NLU) engine processes that text. If the ASR makes a mistake, the NLU is likely to fail.
  2. Data Scarcity: Labeling speech with intents and slots is incredibly expensive and time-consuming, leading to small, "clean" datasets that don't reflect real-world usage.
  3. Environmental Noise: Most benchmarks use pristine audio, but real-world assistants must operate in noisy environments where standard models often break down.
Comparison of SLU paradigms, highlighting the proposed semi-supervised approach (D) that utilizes both labeled and unlabeled audio.

Methods

The authors propose an end-to-end (E2E) framework that bridges the gap between acoustic signals and semantic meaning.

  • Model Architecture: They combine a Transformer-based E2E ASR model with a pretrained BERT. To make them work together, they developed a method to align the different subword tokenizations used by each model (BPE for ASR vs. WordPiece for BERT) by mapping hidden states into a shared matrix before concatenation.
  • Two-Stage Fine-Tuning: They first pretrain the ASR component on a large speech corpus (Librispeech) and then jointly fine-tune the entire ASR+BERT stack on the target SLU data.
  • Semi-Supervised Learning: They explore two settings for when labeled data is scarce:
  • Supervised Pretraining: Using external transcribed speech to "warm up" the ASR.
  • Unsupervised Pretraining: Replacing the ASR encoder with wav2vec features, allowing the model to learn from raw audio without any transcripts.
  • Noise Augmentation: To handle real-world conditions, they train the model using the MS-SNSD dataset, which injects various environmental noises into the training audio.
  • New Metric: They introduce slots edit $F_1$, a more rigorous evaluation metric that accounts for instances where the predicted word sequence length differs from the ground truth—a common issue in E2E systems.
Detailed architecture of the proposed framework, integrating E2E ASR and BERT for joint Intent Classification (IC) and Slot Labeling (SL).

Datasets

  • ATIS (Airline Travel Information System): The primary benchmark for SLU, consisting of ~8 hours of flight reservation queries.
  • Librispeech (LS-960): 960 hours of read speech used for pretraining the ASR component.
  • MS-SNSD: A dataset of environmental noises used for data augmentation.
  • WSJ (Wall Street Journal): Used as a baseline for prior work comparisons.

Key Insights and Ideas

  • Closing the "Oracle" Gap: The most striking finding is that an E2E model taking raw audio as input can perform nearly as well as an NLU model taking perfect human-transcribed text. This suggests that joint fine-tuning allows the model to recover from ASR errors.
  • Context Matters in wav2vec: When using unsupervised pretraining, the authors found that wav2vec’s context vector ($c$) is far superior to the latent vector ($z$) for semantic tasks. Adding a subsampling layer to handle the high frame rate of these features was critical for success.
  • The Necessity of Noise Training: Models trained only on "clean" data saw a massive performance drop (up to 27%) when tested on noisy audio. However, with noise augmentation, the E2E model was remarkably resilient, outperforming traditional hybrid systems by nearly 40% in noisy conditions.

Key Results

  • Clean Performance: On the ATIS dataset, the proposed E2E model achieved 95.88% slots edit $F_1$ and 97.26% intent $F_1$, essentially matching the performance of the "Oracle Text" baseline (95.64% / 98.99%).
  • Noisy Performance: In noisy environments, the model maintained a high 95.46% SL $F_1$, whereas a standard Kaldi-based hybrid system plummeted to 69.55%.
  • Unsupervised Success: Using wav2vec 1.0 with subsampling reached 85.64% SL $F_1$ without using a single transcript during the initial speech pretraining phase, proving the viability of SLU for low-resource languages.

Limitations and Future Work

While the results on ATIS are impressive, the authors note that ATIS is a relatively "shallow" and structured dataset. Future work should explore:

  • Spontaneous Speech: Moving beyond short commands to handle long-form, conversational, and spontaneous speech.
  • Multi-lingual SLU: Expanding the framework to support multiple languages simultaneously.
  • Model Efficiency: Exploring how to make these large ASR+BERT stacks efficient enough to run on edge devices (like smart speakers) rather than just in the cloud.

Summary updated

Abstract

Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.
BibTeX
@inproceedings{lai2021semi,
  author = {Cheng-I Lai and Yung-Sung Chuang and Hung-yi Lee and Shang-wen Li and James R. Glass},
  title = {{Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414922},
}
Figure