TL;DR
This paper tackles the challenge of ambiguous utterances in intelligent assistants, where user intent can be unclear (e.g., "I'm hungry" could be a request for food or a casual statement). They introduce a novel approach using sentence embeddings from external knowledge sources and a self-attention mechanism to accurately detect these ambiguous intents, outperforming strong baselines.
What Problem Does This Paper Address?
Intelligent assistants like Siri and Alexa are increasingly sophisticated, handling both task-oriented requests (e.g., "Set a timer") and casual conversation (chat). A significant challenge arises when users utter ambiguous phrases, where the assistant struggles to discern whether the user wants to perform a task or simply engage in conversation. For example, "Tokyo station" could mean a train route search or a general web search, while "I'm hungry" could be a request for restaurant information or a simple statement of a physical state. Naively interpreting these ambiguous utterances can lead to mismatched responses and a poor user experience. Existing methods often focus on task-oriented dialogues or struggle to adapt to the hybrid nature of intelligent assistants, which mix task and chat intents in short, often noisy utterances. This paper aims to robustly detect these ambiguous utterances in intelligent assistants.
Methods
The core of the proposed method lies in leveraging external knowledge and a sophisticated attention mechanism to enhance intent detection. The authors first construct a dataset of ambiguous utterances from real-world intelligent assistant logs. They then employ a BERT-based classifier, but with a crucial enhancement: they feed sentence embeddings derived from two distinct sources into the model. These sources are:
- Search query logs: To capture task-oriented intents.
- Microblog logs (X logs): To capture chat-oriented intents.
These sentence embeddings are generated using SimCSE, a contrastive learning method known for improving sentence representation. Crucially, a self-attention mechanism is applied to these embeddings (along with the BERT output) to weigh their importance and capture their interrelationships. This allows the model to consider the relevance of both task-specific and chat-specific knowledge when determining the intent of an utterance.
Datasets
- Yahoo! Voice Assist Logs: The authors collected 20,000 Japanese conversations from Yahoo! Voice Assist, a commercial Japanese intelligent assistant, spanning from 2014 to 2022. These logs were then crowdsourced to label utterances as 'chat,' 'task,' or 'ambiguous.' This forms a new benchmark dataset for ambiguous utterance detection in intelligent assistants.
- External Datasets for Embeddings:
- 50 million top-frequent Japanese web search queries (July 2021-July 2022).
- 50 million randomly sampled Japanese tweets.
Key Insights and Ideas
- Ambiguity is prevalent and challenging: The dataset analysis revealed that the 'ambiguous' label was the most frequent, highlighting the significant challenge of intent disambiguation in intelligent assistants.
- Short, noisy utterances are key: The paper emphasizes that the brevity and potential noise in intelligent assistant utterances make them particularly difficult to classify, often obscuring intent.
- External knowledge is vital: Simply relying on dialogue context is insufficient. Incorporating knowledge from search queries (task) and microblogs (chat) significantly improves the ability to disambiguate intents.
- Self-attention is powerful: The self-attention mechanism effectively weighs the importance of different embeddings, allowing the model to better understand the nuances of ambiguous utterances.
- LLMs might struggle with inherent ambiguity: The GPT-4o results suggest that even powerful LLMs can find it difficult to grasp the concept of ambiguity without specific external knowledge or context.
Key Results
The proposed method, leveraging external sentence embeddings and self-attention, achieved the highest performance across all metrics, including F1-score.
- Overall F1-score: The proposed method achieved an F1-score of 79.31%, outperforming baselines like SVM (76.72%), BERT (base) (77.48%), and BERT (large) (77.63%).
- Ambiguous Label F1-score: Notably, the proposed method showed a significant gain in detecting the 'ambiguous' label, exceeding 3% compared to other models, demonstrating its effectiveness in identifying uncertain intents.
- Qualitative Analysis: The model was particularly adept at detecting utterances with speech recognition errors and correctly disambiguating utterances that could be interpreted as either a command or a casual expression.
Limitations and Future Work
- Limitations:
- The paper notes that even with the proposed method, distinguishing between utterances that are solely a single noun (e.g., "Shinagawa") and those that are clearly unambiguous (e.g., "Lexus") remains challenging. This suggests a need for more detailed external knowledge about specific entities.
- The performance of GPT-4o was surprisingly lower than expected, suggesting LLMs might struggle with inherent ambiguity without further fine-tuning or specific prompting strategies.
- Future Work:
- The authors plan to integrate their ambiguity detection method with a module for generating clarification questions to further improve user experience.
- They aim to gather feedback from users to refine the system.
- The newly created dataset will be released to facilitate future research in this area.
Summary updated