TL;DR
This paper introduces a novel zero-shot intent classification approach using Long-Short Term Memory (LSTM) networks. By learning semantic similarities between user queries and intents, the model can accurately identify intents it has never encountered during training, significantly outperforming previous methods. This advancement is crucial for building more adaptable and scalable intelligent assistant systems.
What Problem Does This Paper Address?
Intelligent Assistant (IA) systems, like Siri or Cortana, rely heavily on Spoken Language Understanding (SLU) to interpret user requests. A core component of SLU is intent classification, where the system determines the user's goal (e.g., "set an alarm," "book a hotel"). Traditionally, IA systems are trained on a predefined set of intents. When a new, unseen intent emerges (e.g., "book a taxi to the hotel" when only hotel booking intents were trained), the system fails. Redesigning the entire system schema and retraining is time-consuming and expensive. This paper tackles the challenge of enabling IA systems to classify intents they haven't been explicitly trained on, a problem known as zero-shot intent classification.
Methods
The core of the proposed solution is a Long-Short Term Memory (LSTM) neural network used as an encoder. This encoder takes both user utterances and intent labels as input and generates semantic embeddings for each. The intuition is that utterances and their corresponding intents will have similar embeddings in this learned semantic space.
The LSTM encoder is designed to process sequential data effectively, leveraging both character-level and word-level bidirectional LSTMs. This allows it to capture rich semantic nuances from the input text.
For classification, the model computes the cosine similarity between the utterance's embedding and the embeddings of all potential intents (both seen and unseen). The intent with the highest similarity score is predicted. The probability of an intent given an utterance is then derived using a softmax function over these similarity scores.
The paper also explores a few-shot learning scenario, where a small number of training samples for new intents are available. In this setting, the pre-trained zero-shot model is fine-tuned with these new samples.
Datasets
The experiments were conducted on a dataset of nearly 200,000 utterances from a commercial Intelligent Assistant. This dataset comprises 119 unique intents, typically 1-3 words long, covering common IA domains like weather, reminders, and email.
The data was partitioned based on intents to ensure no overlap between training, validation, and test sets for the zero-shot experiments. Specifically:
- Training set: 154,072 utterances from 89 intents.
- Validation set: 20,915 utterances from 10 intents (not in training).
- Test set: 20,000 utterances from 19 intents (not in training).
For the "previously seen intents" experiment, a separate test set of 19,260 utterances with the same 89 intents as the training set was used.
Key Insights and Ideas
- LSTM's Sequential Power for Semantics: The paper strongly advocates for LSTMs over simpler feed-forward or convolutional architectures for capturing the sequential and contextual semantics inherent in both natural language utterances and intent labels.
- Semantic Space for Generalization: By mapping utterances and intents into a shared semantic space, the model can infer relationships between unseen intents and utterances based on their proximity, enabling zero-shot classification.
- Few-Shot Learning as a Practical Extension: The research demonstrates that even a small number of examples for new intents can significantly boost performance, making the system more robust in real-world scenarios where new intents gradually emerge.
- Similarity Models Shine When Data is Scarce: While standard supervised classifiers excel with abundant data for known classes, similarity-based models like the proposed LSTM encoder become more advantageous when dealing with new, unseen classes or limited data.
Key Results
- Zero-Shot Performance: The proposed LSTM Encoder achieved an accuracy of 43.83% on previously unseen intents, a significant improvement over the Deep Semantic Similarity Model (DSSM) baseline, which achieved 28.60%. The standard LSTM Softmax model (trained supervisedly) achieved 0% accuracy on unseen intents, as expected.
- Few-Shot Performance: In the few-shot setting, accuracy increased with more training examples. For instance, the LSTM Encoder's accuracy rose from 43.83% (zero-shot) to 66.20% with 10 training examples per new intent. This consistently outperformed the DSSM.
- Previously Seen Intents: For classifying intents seen during training, the standard LSTM Softmax model achieved the highest accuracy (81.87%), highlighting its effectiveness when ample labeled data is available for all classes. However, the similarity-based models still performed reasonably well.
Limitations and Future Work
- Dataset Specificity: The experiments were conducted on a specific dataset from a commercial IA. The generalizability of the model to other domains or datasets with different intent structures might need further investigation.
- Computational Cost: While not explicitly detailed as a limitation, training and encoding with LSTMs can be computationally intensive compared to simpler models.
- Future Work: The authors suggest that this work provides a step towards training zero-shot models for SLU systems. Future research could explore more sophisticated ways to leverage semantic information, potentially incorporating external knowledge bases or more advanced attention mechanisms to further improve zero-shot and few-shot performance. The paper also hints at the potential for online adaptation strategies for continuously updating the model with new intents.
Summary updated