Abstract
We present a comprehensive study on building and adapting RNN transducer (RNN-T) models for spoken language understanding (SLU). These end-to-end (E2E) models are constructed in three practical settings: a case where verbatim transcripts are available, a constrained case where the only available annotations are SLU labels and their values, and a more restrictive case where transcripts are available but not corresponding audio. We show how RNN-T SLU models can be developed starting from pre-trained automatic speech recognition (ASR) systems, followed by an SLU adaptation step. In settings where real audio data is not available, artificially synthesized speech is used to successfully adapt various SLU models. When evaluated on two SLU data sets, the ATIS corpus and a customer call center data set, the proposed models closely track the performance of other E2E models and achieve state-of-the-art results.
Introduction
Traditional Spoken Language Understanding (SLU) systems typically follow a "cascaded" architecture, where an Automatic Speech Recognition (ASR) engine converts speech to text, followed by a Natural Language Understanding (NLU) module that extracts semantic meaning. This approach is prone to error propagation, where ASR inaccuracies directly degrade NLU performance.
End-to-end (E2E) models offer a promising alternative by mapping audio directly to semantic labels. However, they are often hindered by the scarcity of high-quality datasets containing aligned audio, transcripts, and semantic labels. This research explores the application of the RNN-Transducer (RNN-T) architecture to SLU across three realistic data-constrained scenarios:
- [FULL]: Audio + Transcripts + SLU Labels.
- [AUDIO]: Audio + SLU Labels (no verbatim transcripts).
- [TEXT]: Transcripts + SLU Labels (no original audio, often due to privacy constraints).
Methodology
The core of the proposed system is the RNN-Transducer (RNN-T), which integrates a transcription network (acoustic encoder), a prediction network (language model), and a joint network. To address data scarcity, several strategies are employed:
- Bootstrapping and Pre-training: Models are initialized using ASR models pre-trained on large, task-independent datasets (e.g., Switchboard).
- Vocabulary Expansion: The RNN-T output layer is expanded to include non-acoustic symbols representing semantic entities and intent labels.
- Synthetic Audio Generation: For the [TEXT] setting, Text-to-Speech (TTS) systems generate training data. The study compares single-speaker (SS) vs. multi-speaker (MS) TTS approaches.
- Curriculum Learning: Models transition from general ASR to domain-specific ASR, and finally to the target SLU task.
Experimental Setup
The models were evaluated on two benchmarks:
- ATIS (Air Travel Information System): A classic dataset (~9.6 hours) for flight reservations, involving slot filling and intent recognition.
- Call Center (CC): A more challenging internal IBM dataset (~17.5 hours) featuring spontaneous telephony speech from customer service calls.
Results and Discussion
The RNN-T models achieved state-of-the-art results, matching or exceeding previous attention-based E2E models. On the ATIS dataset, the model reached an F1 score of 93.2 for entity recognition and 94.7% accuracy for intent recognition.
Performance Scaling
As shown in Figure 1, pre-training on large-scale ASR data is a critical factor for performance. Even with 0 hours of pre-training, the model achieves respectable results, but performance scales significantly as pre-training data increases to 300 hours, particularly for the more complex Call Center dataset.
An important observation is that the adaptation step is highly effective; the final SLU performance is not strictly limited by the initial accuracy of the base ASR model. Furthermore, the models demonstrated implicit adaptation: when trained for SLU with full transcripts, the underlying word recognition (WER) improved without a separate ASR adaptation step.
Key Insights
The Importance of Label Order and Acoustic Diversity
A key finding is the sensitivity of RNN-T to the order of semantic labels. In the [AUDIO] setting, performance is significantly higher when entities are provided in the order they were spoken (spoken-order) rather than alphabetically (alphabetical-order).
Additionally, for the [TEXT] setting, acoustic diversity in synthetic data proved more important than fidelity. Using multi-speaker (MS) TTS to adapt the entire network was significantly more effective than single-speaker (SS) TTS, as it prevented the transcription network from over-fitting to a single synthetic voice. As seen in Figure 5, the MS approach ([4T]) achieved an F1 of 93.2, matching the performance of models trained on real audio.
Conclusion
This study demonstrates that RNN-T models can be successfully adapted for SLU even under severe data constraints. While the models perform exceptionally well when labels follow spoken order, future work is needed to address the "bag-of-entities" problem where label order is arbitrary. Additionally, while multi-speaker TTS narrows the gap, a performance difference remains on complex real-world datasets like the Call Center corpus, highlighting the need for further improvements in synthetic data naturalness and diversity.
Summary updated