RNN Transducer Models for Spoken Language Understanding

Samuel Thomas, Hong-Kwang Jeff Kuo, George Saon, Zoltán Tüske, Brian Kingsbury, Gakuto Kurata, Zvi Kons, Ron Hoory

TL;DR

IBM researchers have successfully adapted the RNN-Transducer (RNN-T) architecture—originally designed for streaming speech recognition—to perform end-to-end Spoken Language Understanding (SLU). Their approach achieves state-of-the-art results across various data-constrained scenarios, including cases where verbatim transcripts are missing or where only synthetic audio is available.

What Problem Does This Paper Address?

Traditional Spoken Language Understanding (SLU) systems are usually "cascaded": an Automatic Speech Recognition (ASR) engine converts speech to text, and a Natural Language Understanding (NLU) module extracts meaning (intents and entities). This approach suffers from error propagation—if the ASR mishears a word, the NLU is likely to fail.

End-to-end (E2E) models aim to map audio directly to semantic labels, but they face a major hurdle: data scarcity. High-quality datasets containing audio, verbatim transcripts, and semantic labels are rare. This paper explores how to build robust RNN-T models in three realistic, constrained settings:

  1. [FULL]: Audio + Transcripts + SLU Labels.
  2. [AUDIO]: Audio + SLU Labels (no verbatim transcripts).
  3. [TEXT]: Transcripts + SLU Labels (no original audio, often due to privacy restrictions).

Methods

The authors utilize the RNN-Transducer (RNN-T), which consists of three sub-networks: a transcription network (acoustic encoder), a prediction network (language model-like), and a joint network that combines them.

To overcome data limitations, they employ several clever strategies:

  • Bootstrapping: They initialize the SLU models using ASR models pre-trained on large, task-independent datasets (like Switchboard).
  • Vocabulary Expansion: The output layer of the RNN-T is expanded to include non-acoustic symbols representing semantic entities (e.g., B-toloc.city_name) and intent labels.
  • Curriculum Learning: In the [FULL] setting, they transition the model from general ASR to domain-specific ASR, and finally to the SLU task.
  • Synthetic Audio for [TEXT] Settings: When real audio is unavailable, they use Text-to-Speech (TTS) systems to generate training data. They specifically compare single-speaker vs. multi-speaker TTS.
  • Pseudo-SLU Pre-training: They hypothesize that training the ASR model to recognize Part-of-Speech (POS) tags helps the network learn to predict abstract, non-acoustic symbols before it ever sees a real SLU label.

Datasets

The models were evaluated on two distinct benchmarks:

  • ATIS (Air Travel Information System): A classic SLU dataset (~9.6 hours of audio) involving flight reservations. It includes both slot filling (entities) and intent recognition.
  • Call Center (CC): An internal IBM dataset consisting of real-world, spontaneous customer service calls (~17.5 hours). This dataset is more challenging due to the nature of telephony speech and varied customer expressions.

Key Insights and Ideas

  • Pre-training is Essential, but Perfection Isn't: While bootstrapping from an ASR model is critical, the initial accuracy of that ASR model doesn't strictly dictate the final SLU performance. The adaptation step is powerful enough to "fix" a mediocre base model.
  • Order Matters for RNN-T: In the [AUDIO] setting (labels only), the model performs significantly better if entities are provided in the order they were spoken. If entities are sorted alphabetically (a "bag-of-entities" approach), performance drops from 93.9 to 79.4 F1 on ATIS.
  • Diversity Trumps Fidelity in TTS: For the [TEXT] setting, using multi-speaker TTS to adapt the entire network is much more effective than using a single-speaker TTS. This prevents the transcription network from over-fitting to the specific acoustic quirks of one synthetic voice.
  • Implicit Adaptation: The authors found that when a model is adapted to SLU with full transcripts, it implicitly improves its underlying word recognition (WER) without needing a separate ASR adaptation step.

Key Results

  • State-of-the-Art Performance: On the ATIS dataset, the RNN-T model achieved an F1 score of 93.2 for entity recognition and 94.7% accuracy for intent recognition, matching or exceeding previous attention-based E2E models.
  • Robustness in [AUDIO] Setting: Even without verbatim transcripts, the model achieved 94.7% intent accuracy on ATIS, showing that RNN-T can learn semantic mappings directly from audio.
  • Success with Synthetic Data: In the [TEXT] setting, using multi-speaker TTS allowed the model to reach an F1 of 93.2 on ATIS, effectively closing the gap with models trained on real human speech.
  • Call Center Challenges: On the more difficult Call Center dataset, the model achieved 87.4% intent accuracy using real audio. Using multi-speaker TTS in the [TEXT] setting reached 83.5%, a respectable result given the acoustic mismatch.

Limitations and Future Work

  • The "Bag-of-Entities" Weakness: RNN-T models struggle when semantic labels are not provided in chronological order. Future work could explore architectural tweaks to make the model less sensitive to label ordering.
  • The Reality Gap: There remains a ~4% performance gap between models trained on real audio versus synthetic audio on complex datasets. Improving the "naturalness" and acoustic diversity of TTS data remains a priority.
  • POS Tagging Mixed Results: While POS tags helped on the ATIS task, they didn't provide a clear benefit for the Call Center task, suggesting that the utility of "pseudo-SLU" pre-training may be domain-dependent.

Summary updated

Abstract

We present a comprehensive study on building and adapting RNN transducer (RNN-T) models for spoken language understanding (SLU). These end-to-end (E2E) models are constructed in three practical settings: a case where verbatim transcripts are available, a constrained case where the only available annotations are SLU labels and their values, and a more restrictive case where transcripts are available but not corresponding audio. We show how RNN-T SLU models can be developed starting from pre-trained automatic speech recognition (ASR) systems, followed by an SLU adaptation step. In settings where real audio data is not available, artificially synthesized speech is used to successfully adapt various SLU models. When evaluated on two SLU data sets, the ATIS corpus and a customer call center data set, the proposed models closely track the performance of other E2E models and achieve state-of-the-art results.
BibTeX
@inproceedings{thomas2021rnn,
  author = {Samuel Thomas and Hong-Kwang Jeff Kuo and George Saon and Zoltán Tüske and Brian Kingsbury and Gakuto Kurata and Zvi Kons and Ron Hoory},
  title = {{RNN Transducer Models for Spoken Language Understanding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=SAebD-CCIl5},
}

Loading related content...

Figure