TL;DR
This paper introduces an end-to-end (E2E) speech-to-intent (S2I) model for e-commerce customer support voicebots, significantly outperforming traditional multi-component pipelines. By leveraging pre-trained Automatic Speech Recognition (ASR) models and fine-tuning them for intent classification, they achieve higher accuracy and lower latency with substantially less annotated data.
What Problem Does This Paper Address?
Customer support voicebots are crucial for handling customer queries efficiently. Traditionally, these voicebots rely on a pipeline of separate Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) modules. This multi-component approach, while functional, suffers from several drawbacks:
- Error Propagation: Errors made by the ASR module (e.g., misrecognizing words) are passed to the NLU module, leading to compounding errors and reduced overall performance.
- High Latency: Each component in the pipeline adds processing time, increasing the overall response latency.
- Complex Deployment and Maintenance: Managing and optimizing multiple independent models is challenging.
- Data Requirements: Training each component often requires large, independently annotated datasets, which are expensive and time-consuming to create.
The paper aims to address these limitations by proposing a unified, end-to-end model that directly maps speech to intent, bypassing the intermediate text transcription step.
Methods
The core innovation lies in adapting a pre-trained end-to-end ASR model to perform intent classification. Instead of a separate ASR and NLU module, they propose an E2E Speech-to-Intent (S2I) model.
- E2E ASR Model: They utilize a pre-trained ASR model based on BiLSTM and CTC (Connectionist Temporal Classification) with attention mechanisms. This model is designed to predict characters, short subwords, and long subwords.
- Intent Prediction Head: A novel "intent prediction head" is added to the ASR model. This head takes the hidden features from the last block of the ASR model and uses a multi-headed self-attention (MHA) layer to summarize these features. A linear layer then predicts the probability distribution over the intent classes.
- Fine-tuning: The pre-trained ASR model is fine-tuned for the S2I task. This involves training the intent prediction head and slightly adjusting the BiLSTM blocks using cross-entropy loss.
- Bilingual Support: The approach is designed to work for both Hindi and English, leveraging a bilingual ASR system.
This E2E approach eliminates the need for a separate ASR decoder and NLU module, simplifying the architecture and reducing error propagation.
Datasets
The paper utilizes several datasets for training and evaluation:
- ASR Training: A large collection of datasets including Flipkart customer support voicebot queries, voice search queries, and general domain speech data. This amounts to approximately 11 million audio-text pairs (~17k hours of audio) in a mix of Hindi and English.
- Transliteration: A dataset of ~96k manually transliterated unique words for Hindi and English.
- Text-to-Intent (Baseline): ~90k manually labeled text-intent pairs from customer support voicebot and chatbot queries.
- Speech-to-Intent (E2E):
- V1: 10k randomly sampled voicebot queries, manually labeled for intents.
- V2: An expanded set of 10k + 25k audios, where the additional 25k were obtained through offline active learning (identifying low-confidence predictions on unlabeled data and manually correcting them).
- Pseudo-labeled data: The model was further trained on ~920k audios from the ASR dataset using pseudo labels.
The paper highlights that the training datasets for ASR and Text-to-Intent in the baseline pipeline had limited overlap, necessitating the creation of smaller, independently labeled datasets for the E2E S2I model.
Key Insights and Ideas
- Leveraging ASR Pre-training for S2I: The most significant insight is that a pre-trained ASR model can be effectively adapted for intent classification with minimal modifications and significantly less labeled data. This is because the ASR model already learns to process acoustic features and extract relevant information from speech.
- Mitigating Error Propagation: By directly mapping speech to intent, the E2E model inherently avoids the error propagation issues that plague traditional ASR-NLU pipelines.
- Data Efficiency: The E2E approach demonstrates remarkable data efficiency. A model fine-tuned on just 10k labeled audio-intent pairs (V1) already outperforms the baseline. Further improvements are achieved with an additional 25k samples.
- Importance of Initialization: Training the S2I model from scratch is suboptimal, underscoring the critical role of initializing the network with ASR pre-trained weights, especially in low-resource scenarios.
- Acoustic and Textual Noise Resilience: The E2E model shows improved performance even when the input data contains acoustic and textual noise, which is common in real-world voicebot scenarios.
Key Results
- Performance Improvement: The final E2E S2I model outperforms the conventional pipeline by an absolute 4.59% on the F1 score. A preliminary version (V1) trained on only 10k pairs achieved a 3.07% absolute improvement.
- Latency Reduction: The E2E model achieves a median latency of ~41ms, which is 1/3rd of the baseline latency (~123ms).
- Scalability: The E2E model can handle over 1000 queries per second on a single A100 GPU, significantly higher than the baseline's ~90 queries per second.
- ASR Performance: The baseline ASR system has a Word Error Rate (WER) of 8.34%, and the ASR + transliteration system has a WER of <9.2%. The text-to-intent model achieves an F1 score of 85.84% with manual transcriptions.
- E2E vs. Text-to-Intent: The E2E model is not only better than the full pipeline but also surpasses the standalone text-to-intent model (with manual transcriptions) by 1.53% on the F1 score.
Limitations and Future Work
- Dataset Mismatch: The baseline text-to-intent model was trained on a different dataset than the one used for fine-tuning the E2E models. While the E2E models still outperform, this difference in dataset size and composition could be a disadvantage.
- Transliteration Module: The baseline pipeline requires a transliteration module due to script mismatches, which adds complexity. However, the authors note its high accuracy (<1% WER) should not significantly impact the baseline.
- ASR Pre-training Data: The large dataset required for ASR pre-training might not be readily available for low-resource languages.
- Multi-lingual Extension: The authors suggest exploring how these models would perform with more than two languages, especially in low-resource scenarios. They hypothesize that adapted S2I models could offer even greater advantages by reducing vocabulary size when intent categories are shared across languages.
Summary updated