MTL-SLT: Multi-Task Learning for Spoken Language Tasks

Zhiqi Huang, Milind Rao, Anirudh Raju, Zhe Zhang, Bach Bui, Chul LeeConvAI @ ACL 2022 · OpenReview

TL;DR

This paper introduces MTL-SLT, a novel multi-task learning framework that jointly trains Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) models for spoken language tasks. By sharing parameters and optimizing for a combined sequence loss, MTL-SLT achieves state-of-the-art results on spoken language understanding tasks and improves robustness to ASR errors.

What Problem Does This Paper Address?

Voice-controlled applications are increasingly common, driving demand for systems that can understand spoken language. Traditionally, speech recognition (ASR) and natural language processing (NLP) have been treated as separate problems, or NLP tasks have been trained independently on transcribed speech. This separation leads to a performance bottleneck, as errors in ASR directly impact downstream NLP tasks. Furthermore, existing multi-task learning (MTL) approaches often focus on either text-based NLP or speech-based tasks, but not the joint optimization of ASR and multiple downstream NLP tasks. This paper aims to bridge this gap by proposing a unified framework that leverages shared representations and joint training to improve performance and robustness across a suite of spoken language tasks.

Methods

The core of MTL-SLT is a multi-task learning framework that integrates an ASR model (specifically, a Listen, Attend and Spell (LAS) model) and an NLP model (BART, a pre-trained encoder-decoder model). The framework operates in three stages:

  1. Backbone Pre-training: The ASR model is pre-trained on the LibriSpeech dataset, and the BART NLP model is pre-trained for text reconstruction.
  2. MTL Pre-training: The ASR and NLP models are jointly trained on multiple spoken language tasks, including summarization, intent classification, slot filling, and question answering. This stage leverages shared parameters between the ASR and NLP components.
  3. Joint Training with Sequence Loss: A crucial innovation is the introduction of a sequence loss that allows for end-to-end optimization of both ASR and NLP components. This sequence loss enables direct optimization of non-differentiable metrics like Word Error Rate (WER) for ASR and task-specific metrics (e.g., ROUGE for summarization, exact match for QA) for NLP. This joint training approach allows gradients to flow from the NLP tasks back to the ASR model, making the system more robust to ASR errors.

Datasets

The paper evaluates MTL-SLT on four datasets:

  • ATIS: A widely used dataset for spoken language understanding in airline reservation systems, focusing on intent detection and slot filling.
  • SLURP: A larger and more complex spoken language understanding dataset with 18 domains.
  • Spoken-SQUAD: An extraction-based spoken question answering dataset.
  • Spoken-Gigaword: A newly prepared dataset for abstractive summarization of speech, generated from the Gigaword dataset using a text-to-speech model.

Key Insights and Ideas

  • Joint ASR-NLP training is crucial for robustness: By jointly training the ASR and NLP models with a sequence loss, the system becomes significantly more robust to errors introduced by the ASR component. This is a key differentiator from pipeline approaches.
  • Multi-task learning benefits both ASR and NLP: The MTL framework not only improves performance on individual NLP tasks but also shows advantages for the ASR task itself, suggesting that learning from diverse spoken language tasks can lead to better speech recognition representations.
  • Sequence loss enables end-to-end optimization of non-differentiable metrics: The ability to optimize for metrics like WER directly, rather than relying solely on differentiable cross-entropy losses, is a powerful mechanism for improving performance on real-world spoken language tasks.
  • Pre-trained models are essential: Leveraging pre-trained LAS for ASR and BART for NLP provides a strong foundation, and the MTL framework further enhances these models by training them jointly on related tasks.

Key Results

The paper reports state-of-the-art results on spoken language understanding tasks, specifically on the ATIS and SLURP datasets. For example, their MTL-SLT model achieved an accuracy of 98.18% and an F1 score of 96.51% on the ATIS test set, outperforming previous methods. They also demonstrate improvements in ASR metrics (WER) and NLP metrics across various tasks when using their jointly trained MTL-SLT model compared to independent or pipeline approaches. The paper also highlights that their jointly trained models (M+M in Table 2) perform better than pipeline models (S+S in Table 2), underscoring the benefits of end-to-end joint training.

Limitations and Future Work

The paper acknowledges that while their framework extends to multiple NLP tasks, further research could explore its application to other language tasks, such as translation. The authors also make their code and data publicly available, encouraging further research in this area. The experimental results suggest that the proposed MTL-SLT framework is a promising direction for building more robust and performant spoken language understanding systems.

Summary updated

BibTeX
@inproceedings{huang2022mtl,
  author = {Zhiqi Huang and Milind Rao and Anirudh Raju and Zhe Zhang and Bach Bui and Chul Lee},
  title = {{MTL-SLT: Multi-Task Learning for Spoken Language Tasks}},
  booktitle = {ConvAI @ ACL},
  year = {2022},
  url = {https://openreview.net/forum?id=_uk3kzZJ8ls},
}
Figure