A Streaming End-to-End Framework For Spoken Language Understanding

Nihal Potdar, Anderson Raymundo Avila, Chao Xing, Dong Wang, Yiran Cao, Xiao ChenIJCAI 2021 · DOI

TL;DR

This paper introduces a novel streaming end-to-end framework for Spoken Language Understanding (SLU) that can process multiple user intents incrementally and in real-time. By leveraging a unidirectional RNN trained with Connectionist Temporal Classification (CTC), the system achieves high accuracy comparable to state-of-the-art non-streaming models while significantly reducing latency and enabling more natural, multi-intent interactions.

What Problem Does This Paper Address?

Traditional Spoken Language Understanding (SLU) systems often rely on a "tandem" approach, where speech is first recognized into text (ASR), and then that text is interpreted to understand the user's intent (NLU). This modular approach suffers from error propagation between modules and is not globally optimal. More recent "end-to-end" SLU systems aim to directly extract intents from speech, offering joint optimization and lower latency. However, existing end-to-end SLU systems typically process only one intent at a time and require the entire utterance to be processed before making a prediction. This leads to significant latency, frustrating user experiences, and necessitates multiple interaction rounds for users with complex requests. The paper addresses the critical need for SLU systems that can handle multiple intents simultaneously, in real-time, and without waiting for the end of an utterance.

Methods

The core of the proposed framework is a unidirectional Recurrent Neural Network (RNN), specifically using Long Short-Term Memory with Projection (LSTMP) units. A key innovation is the use of Connectionist Temporal Classification (CTC) for training. CTC is crucial because it allows the model to learn alignments between input speech features and output intent labels without requiring explicit segmentation. This enables the model to predict an intent only when sufficient evidence has been accumulated, rather than at every time step.

To further improve performance and address data sparsity, a two-stage pre-training strategy is employed:

  1. ASR Pre-training: The initial layers of the RNN are initialized with weights from a pre-trained ASR model. This leverages large speech corpora to learn robust acoustic feature extraction.
  2. Cross-Entropy (CE) Pre-training: After ASR pre-training, the model is trained with a cross-entropy loss on the intent prediction task. This provides a strong bias for intent alignment before the final CTC training.

The framework is designed to be streaming, meaning it processes audio in chunks and can output predictions incrementally. It is also incremental, allowing it to identify multiple intents sequentially within a single utterance.

Datasets

The paper utilizes several datasets for training and evaluation:

  • LibriSpeech: Used for pre-training the ASR model.
  • Fluent Speech Commands (FSC): A dataset for single-intent classification. The authors also generated new multi-intent datasets (FSC-M2, FSC-M3, FSC-MM) by concatenating utterances from FSC to evaluate multi-intent prediction.
  • Google Speech Commands (GSC) V1 & V2: Used for evaluating the model's capability in spotting short keywords.

Key Insights and Ideas

  • CTC for Streaming SLU: The paper demonstrates that CTC is a powerful tool for enabling streaming SLU. Its ability to output labels only when evidence is sufficient perfectly matches the requirement of identifying intents incrementally.
  • Two-Stage Pre-training is Crucial: Both ASR and CE pre-training significantly boost performance, with CE pre-training showing a particularly strong impact, likely by providing better initial alignment for the CTC loss.
  • Streaming Outperforms Non-Streaming in User Experience: While not directly measured in this paper, the inherent latency reduction of a streaming model is a major advantage for human-computer interaction, as evidenced by prior research on the impact of delays.
  • Multi-Intent Prediction is Challenging but Achievable: The proposed model shows a significant drop in accuracy when directly applied to multi-intent tasks without specific multi-intent training data. However, training on multi-intent data leads to performance comparable to single-intent tasks, highlighting the model's adaptability.
  • Early Spotting is Common and Beneficial: The analysis shows that intents are often spotted well before the end of the utterance, especially in multi-intent scenarios, underscoring the effectiveness of the streaming approach. Longer sentences tend to be spotted earlier.

Key Results

  • Single-Intent Classification: The streaming model achieved an accuracy of 98.90% on the FSC dataset, comparable to state-of-the-art non-streaming models, while being fully streaming.
  • Multi-Intent Prediction: The model demonstrated the ability to predict multiple intents. When trained on multi-intent data, it achieved accuracies of 97.10% (FSC-M3-Tst) and 97.28% (FSC-MM-Tr) for three-intent scenarios, significantly outperforming its performance when trained only on single-intent data (e.g., 26.73% for FSC-M3-Tst).
  • Keyword Spotting: The model achieved competitive performance on the Google Speech Commands dataset, outperforming benchmarks while offering streaming capabilities.
  • Early Spotting: Over 30% of intents were spotted earlier than ground truth in single-intent scenarios, and this rose to 68% in multi-intent scenarios.

Limitations and Future Work

  • Limitations: The paper acknowledges that the multi-intent prediction performance significantly degrades when the model is trained only on single-intent data. This highlights the need for sufficient multi-intent training data for optimal performance.
  • Future Work: The authors propose to explore:
  • Streaming intent and slot classification simultaneously.
  • Integrating local attention mechanisms into the streaming solution.

Summary updated

BibTeX
@inproceedings{potdar2021streaming,
  author = {Nihal Potdar and Anderson Raymundo Avila and Chao Xing and Dong Wang and Yiran Cao and Xiao Chen},
  title = {{A Streaming End-to-End Framework For Spoken Language Understanding}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/538},
}
Figure