End-to-End Spoken Language Understanding Using Transformer Networks and Self-Supervised Pre-Trained Features

Edmilson da Silva Morais, Hong-Kwang Jeff Kuo, Samuel Thomas, Zoltán Tüske, Brian Kingsbury

TL;DR

This paper introduces a modular, Transformer-based architecture for End-to-End (E2E) Spoken Language Understanding (SLU). By combining self-supervised acoustic features (wav2vec) with multi-task learning—predicting both text and semantics simultaneously—the researchers achieved performance that rivals traditional cascaded systems while significantly reducing the need for complex model pre-initialization.

What Problem Does This Paper Address?

Traditional Spoken Language Understanding (SLU) systems are "cascaded": first, an Automatic Speech Recognition (ASR) module turns speech into text, and then a Natural Language Understanding (NLU) module extracts meaning (intents and entities) from that text.

This approach has two major flaws:

  1. Error Propagation: If the ASR makes a mistake, the NLU module is likely to fail because it was trained on "clean" text.
  2. Loss of Information: ASR discards "prosody"—the pitch, speed, and intonation of speech—which often carries vital semantic cues.

While "End-to-End" (E2E) systems aim to solve this by going straight from audio to meaning, they are notoriously difficult to train because labeled SLU data is scarce compared to raw audio or plain text. The authors tackle this by investigating how self-supervised speech features and multi-task training can bridge this gap.

Methods

The authors propose a modular Transformer-based architecture built on the ESPNet toolkit. The core of their approach involves three interconnected tasks:

  1. Speech to Semantics (S2I&E): The main task of mapping audio to intent and entity labels.
  2. Speech to Text (S2T): An auxiliary task that helps the encoder learn phonetic and linguistic representations.
  3. Text to Semantics (T2I&E): An auxiliary task that helps the decoder learn the structure of the semantic output.

Key Innovations: * Feature Extraction: Instead of using standard log-mel filterbanks (spectrograms), they use wav2vec, a self-supervised model pre-trained on massive amounts of unlabeled audio. * Tied Components: To ensure the tasks help each other, the encoders for speech tasks are shared, and the decoders for semantic tasks are shared. * Modular Initialization: The system allows for "pre-initialization," where parts of the network are pre-trained on independent ASR or NLU tasks before the final E2E training begins.

Datasets

  • ATIS (Air Travel Information System): The primary benchmark for SLU, containing queries about flights. The authors used a 16kHz version augmented with speed perturbation to create ~30 hours of training data.
  • LibriSpeech: Used for "enhanced" pre-training of the encoder. They tested subsets of 100 hours and the full 960 hours.

Key Insights and Ideas

  • Self-Supervision is a Power-Up: wav2vec features consistently outperformed traditional filterbanks. In scenarios with no pre-training, wav2vec improved the Entity F1 score from a dismal 34.4% to 76.6%.
  • Multi-tasking as a Shortcut: One of the most significant findings is that combining wav2vec features with multi-task training (predicting text alongside semantics) almost entirely eliminates the need for specialized model pre-initialization. This simplifies the training pipeline significantly.
  • Cross-Modal Benefits: The auxiliary "Text to Semantics" task (which doesn't even use audio) improved the performance of the audio-based model, proving that learning the "language" of the output labels helps the model interpret speech better.

Key Results

The researchers compared their E2E model against two baselines: a "Gold Standard" NLU (using ground-truth text) and a standard Cascaded SLU.

  • Entity Extraction (F1 Score): Their best E2E configuration (using wav2vec and multi-tasking) achieved a 91.4% F1 score, outperforming the Cascaded SLU baseline (88.8%) and coming remarkably close to the NLU-only "Gold Standard" (91.6%).
  • Intent Recognition (IER): The Intent Error Rate dropped to 3.3%, which is essentially on par with the NLU baseline (3.2%).
  • Ensemble Boost: By combining multiple models, they pushed the Entity F1 score even higher to 92.4%, surpassing even the NLU model trained on perfect text.

Limitations and Future Work

The authors acknowledge that they have not yet fully optimized the hyperparameters of their Transformer architecture, suggesting that even better results are possible.

Future directions include: * Testing more recent self-supervised features like wav2vec 2.0, Mockingjay, or HuBERT. * Fine-tuning the self-supervised feature extractors specifically for the SLU task rather than using them as "frozen" features. * Scaling the approach to larger, more complex datasets beyond the relatively small ATIS corpus.

Summary updated

Abstract

Transformer networks and self-supervised pre-training have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of spoken language understanding (SLU) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SLU transformer network based architecture which allows the use of self-supervised pre- trained acoustic features, pre-trained model initialization and multi-task training. Several SLU experiments for predicting intent and entity labels/values using the ATIS dataset are performed. These experiments investigate the interaction of pre-trained model initialization and multi-task training with either traditional filterbank or self-supervised pre-trained acoustic features. Results show not only that self-supervised pre-trained acoustic features outperform filterbank features in almost all the experiments, but also that when these features are used in combination with multi-task training, they almost eliminate the necessity of pre-trained model initialization.
BibTeX
@inproceedings{morais2021end,
  author = {Edmilson da Silva Morais and Hong-Kwang Jeff Kuo and Samuel Thomas and Zoltán Tüske and Brian Kingsbury},
  title = {{End-to-End Spoken Language Understanding Using Transformer Networks and Self-Supervised Pre-Trained Features}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=BgWZPWRRLec},
}

Loading related content...

Figure