End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features

Edmilson da Silva Morais, Hong-Kwang Jeff Kuo, Samuel Thomas, Zoltán Tüske, Brian Kingsbury

Abstract

Transformer networks and self-supervised pre-training have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of spoken language understanding (SLU) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SLU transformer network based architecture which allows the use of self-supervised pre-trained acoustic features, pre-trained model initialization and multi-task training. Several SLU experiments for predicting intent and entity labels/values using the ATIS dataset are performed. These experiments investigate the interaction of pre-trained model initialization and multi-task training with either traditional filterbank or self-supervised pre-trained acoustic features. Results show not only that self-supervised pre-trained acoustic features outperform filterbank features in almost all the experiments, but also that when these features are used in combination with multi-task training, they almost eliminate the necessity of pre-trained model initialization.

Introduction

Traditional Spoken Language Understanding (SLU) systems typically follow a cascaded approach, where an Automatic Speech Recognition (ASR) module converts speech to text, followed by a Natural Language Understanding (NLU) module that extracts semantic meaning. This paradigm suffers from two major flaws: error propagation, where ASR mistakes lead to NLU failures, and the loss of acoustic information such as prosody. While End-to-End (E2E) systems address these issues by mapping audio directly to semantics, they are often difficult to train due to the scarcity of labeled SLU data. This work explores the integration of self-supervised acoustic features and multi-task learning within a Transformer-based framework to bridge this gap.

Methodology

The proposed architecture is a modular Transformer network built on the ESPNet toolkit. It incorporates three primary tasks to leverage both acoustic and linguistic representations:

  1. Speech to Semantics (S2I&E): The primary task of mapping audio to intent and entity labels.
  2. Speech to Text (S2T): An auxiliary task helping the encoder learn phonetic representations.
  3. Text to Semantics (T2I&E): An auxiliary task helping the decoder learn semantic output structures.
The proposed modular E2E SLU transformer architecture featuring shared encoders and decoders for multi-task learning.

Key Innovations

  • Self-Supervised Features: Instead of standard log-mel filterbanks, the model utilizes wav2vec features pre-trained on massive unlabeled audio datasets.
  • Tied Components: Encoders are shared between speech tasks, and decoders are shared between semantic tasks to ensure cross-modal benefits.
  • Modular Initialization: The system allows for pre-initialization of the encoder and decoder using independent ASR or NLU tasks.

Experimental Results

Experiments were conducted using the ATIS dataset, augmented with speed perturbation. The results highlight the transformative impact of self-supervised features and multi-tasking.

Detailed performance metrics (Entity F1 and Intent Error Rate) across various model configurations and feature types.

Impact of wav2vec and Multi-tasking

As shown in the results table, wav2vec features consistently outperformed traditional filterbanks. In a baseline scenario without pre-initialization (Set 1), switching to wav2vec improved the Entity F1 score from 34.4% to 76.6%. Most significantly, combining wav2vec with multi-task training (Set 1, Number 4) achieved an 89.3% F1 score, nearly eliminating the need for specialized pre-initialization.

Comparison with Baselines

The best E2E configuration achieved an Entity F1 score of 91.4%, surpassing the Cascaded SLU baseline (88.8%) and rivaling the "Gold Standard" NLU model (91.6%) which uses ground-truth text. Intent recognition also reached parity with the NLU baseline, with an error rate of 3.3%.

Conclusion

This research demonstrates that modular Transformer architectures utilizing self-supervised features and multi-task learning can achieve state-of-the-art SLU performance. This approach simplifies the training pipeline by reducing the dependency on pre-trained model initialization. Future work will investigate more recent self-supervised features like wav2vec 2.0 and HuBERT, as well as fine-tuning these extractors for specific SLU tasks.

Summary updated

Abstract

Transformer networks and self-supervised pre-training have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of spoken language understanding (SLU) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SLU transformer network based architecture which allows the use of self-supervised pre-trained acoustic features, pre-trained model initialization and multi-task training. Several SLU experiments for predicting intent and entity labels/values using the ATIS dataset are performed. These experiments investigate the interaction of pre-trained model initialization and multi-task training with either traditional filterbank or self-supervised pre-trained acoustic features. Results show not only that self-supervised pre-trained acoustic features outperform filterbank features in almost all the experiments, but also that when these features are used in combination with multi-task training, they almost eliminate the necessity of pre-trained model initialization.
BibTeX
@inproceedings{morais2021end,
  author = {Edmilson da Silva Morais and Hong-Kwang Jeff Kuo and Samuel Thomas and Zoltán Tüske and Brian Kingsbury},
  title = {{End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=BgWZPWRRLec},
}
Figure