TL;DR
This paper introduces LARA, a novel framework that significantly improves multi-turn intent classification for chatbots by leveraging single-turn training data and a retrieval-augmented approach with large language models (LLMs). LARA achieves state-of-the-art performance across six languages, demonstrating its effectiveness in handling complex conversational contexts without requiring extensive multi-turn annotated datasets.
What Problem Does This Paper Address?
Chatbots are increasingly vital for customer service, but accurately understanding user intent in multi-turn conversations remains a significant challenge. Unlike single-turn interactions, multi-turn dialogues require models to consider the entire conversational history, which introduces complexity due to evolving user intentions and nuanced context. The primary hurdle is the difficulty and cost associated with collecting large-scale, multi-turn annotated datasets. Existing methods often rely on such datasets, making them impractical for many real-world applications, especially those with hundreds of intents across multiple languages. This paper aims to bridge this gap by developing a method that can achieve high accuracy in multi-turn intent classification using readily available single-turn data.
Methods
LARA employs a two-pronged strategy:
- Single-Turn Classification Model ($M_c$): The authors first train a specialized single-turn hierarchical text classification (HTC) model. This model is an ensemble of a label-attention mechanism and the HiTIN approach, designed to leverage the hierarchical structure of intents. This $M_c$ is crucial for efficiently identifying a smaller set of candidate intents from the vast number of possible intents.
- Linguistic-Adaptive Retrieval-Augmentation (LARA): This is the core of the proposed framework.
- Candidate Intent Selection: The trained $M_c$ is used to predict a few candidate intents for the current multi-turn query, based on a combination of the current query and historical context.
- In-Context Learning (ICL) Retrieval: For each candidate intent, LARA retrieves semantically similar single-turn examples from a pre-existing single-turn dataset. This retrieval process is guided by embedding similarity.
- Prompt Construction: The retrieved examples, along with a task description and the conversational context, are formatted into a prompt for a large language model (LLM). This prompt guides the LLM to perform zero-shot multi-turn intent classification.
This approach allows LARA to adapt linguistically to different markets and intents without market-specific multi-turn fine-tuning.
Datasets
The paper utilizes a proprietary e-commerce multi-turn dataset collected from eight markets (Brazil, Indonesia, Malaysia, Philippines, Singapore, Thailand, Taiwan, and Vietnam) in their local languages. This dataset consists of user queries and is annotated with the intent of the last query in each conversation session. For training the single-turn model ($M_c$) and for retrieval, the authors leverage abundant single-turn training data from the same e-commerce business operations.
Key Insights and Ideas
- Leveraging Single-Turn Data for Multi-Turn Tasks: The most significant insight is that high-quality multi-turn intent classification can be achieved by cleverly utilizing readily available single-turn data, bypassing the need for expensive multi-turn annotation.
- The Power of Retrieval-Augmented LLMs: LARA demonstrates how retrieval-augmented LLMs can effectively handle complex tasks like multi-turn intent classification in a zero-shot setting, by providing relevant context and examples within the prompt.
- Hierarchical Intent Structure Matters: The use of a hierarchical text classification model ($M_c$) to pre-filter candidate intents is crucial. This pre-filtering step helps manage the large number of intents and reduces noise for the LLM.
- Linguistic Adaptability: The framework's ability to perform well across six different languages without extensive retraining highlights the power of its linguistic-adaptive nature.
Key Results
- LARA achieves state-of-the-art performance on the multi-turn intent classification task, improving average accuracy by 3.67% compared to state-of-the-art single-turn classifiers.
- The framework significantly reduces inference time, improving it by 77% (from 0.75it/s to 1.32it/s on a V100 card).
- Ablation studies show that both the single-turn classification model ($M_c$) and the retrieval mechanism are critical for LARA's performance. Removing $M_c$ led to a significant accuracy drop (from 64.18% to 53.99%), and removing retrievers decreased performance by 0.71%.
- The paper also reports improvements in user satisfaction (1%), reduced transfer-to-agent rates (0.5%), and cost savings for agents in their industrial application.
Limitations and Future Work
- Out-of-Domain Utterances: LARA currently does not address the detection of out-of-domain utterances, which is a critical aspect for robust online dialogue systems.
- Intent Shifts: The resilience of the method to user intent shifts within a conversation has not been examined.
- Implementation Complexity: The multi-component architecture (classification, retrieval, ICL) adds to implementation complexity. The authors propose a lightweight solution for applications with limited deployment resources.
- Future Work: The authors intend to extend LARA to recommendation tasks in other domains, such as understanding user intent shifts in POI or itinerary recommendations, to better capture evolving user preferences. They also suggest further prompt engineering could yield even better results.
Summary updated